[DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
Locked
123
123
Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
 
|
I would say both ways (one unified memory manager per task executor, or one
isolated memory manager per slot) should work well at the moment. I've not yet think this through very detailedly, but just from what I can think of at the moment, I'm in favor of having one memory manager per task executor, for the following reasons: - We need the memory manager to understand quota of each operator anyway (with the provided fractions), for limiting the memory usage of un-revocable operators. It shouldn't be much more complicated for the memory manager to isolate memory across slots. Actually I'm wondering wether it requires more efforts at all compare to having one memory manager per slot, where we also need to handle the creation and destroying of memory managers. - It is true that we have no plan for segments caching at the moment. But I would say it is a good opportunity for future improvement. E.g., we may have some "lazy de-allocation" strategy that only de-allocate segments if the remaining available memory quota cannot satisfy new coming reservation requests. I understand that we should not complicate the current design for any "potential future features". My concern is that we make efforts changing from one memory manager per task executor to one per slot, with the benefit (easier to isolate memory across slots) that is uncertain to me, and later discover that we may want to change it back for further optimizations. Anyway, I do not have strong objection against having one memory manager per slot. Just raising my concerns. Thank you~ Xintong Song On Mon, Sep 16, 2019 at 4:16 PM Andrey Zagrebin <[hidden email]> wrote: > @Xintong > > Just to understand, what kind of overhead do you mean by creating the > memory manager for every dynamic slot? > We agreed not to preallocate segments anymore and there was no plan to do > any segment caching at the moment, AFAIK. > This means the segments are just garbage collected upon release and not > reused. > If we decide to go with caching or revoking later and share pools among > slots, it would already mean mixed segment/reservation types of memory > allocation in general (like batch and streaming). > Then I think memory manager would need some input from tasks because it is > hard for Flink to decide how much to cache and which pool to allocate from > eventually. > As I understand, tasks do not really need this now. That is why I think > having memory manager per slot will give more security by isolating slots > with relatively small effort. > If there is no preallocation/caching then creating the memory manager is > almost no-op, later we can reconsider it if needed. What do you think? > > Thanks, > Andrey > > On Sat, Sep 14, 2019 at 8:11 AM Xintong Song <[hidden email]> > wrote: > > > @Andrey, > > > > If we speak in the condition of dynamic slot allocation, then DataSet > tasks > > should always request slots with unknown resource profiles. Then the > > allocated slots should have 1/n on-heap managed memory and 1/n off-heap > > managed memory of the task executor, where n is the configured > > 'numberOfSlot'. Given that DataSet operators only allocate segments (no > > reservation), and does not care whether the segments are on-heap or > > off-heap, I think it makes sense to wrapping both pools for DataSet > > operators, with respect to the on-heap/off-heap quota of the allocated > > slot. > > > > I'm not sure whether we want to create one memory manager per slot. I > think > > we can have limit the quota of each slot with one memory manager per task > > executor. Given the dynamic slot allocation that dynamically create and > > destroy slots, I think having one memory manager per task executor may > save > > the overhead for frequently creating and destroying memory managers, and > > provide the chance for reusing memory segments across slots. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Fri, Sep 13, 2019 at 11:47 PM Andrey Zagrebin <[hidden email]> > > wrote: > > > > > Hi Xintong, > > > > > > True, there would be no regression if only one type of memory is > > > configured. This can be a problem only for the old jobs running in a > > newly > > > configured cluster. > > > > > > About the pool type precedence, in general, it should not matter for > the > > > users which type the segments have. > > > The first implementation can be just to pull from any pool, e.g. empty > > one > > > pool firstly and then another or some other random pulling. > > > This might be a problem if we mix segment allocations and reservation > of > > > memory chunks from the same memory manager. > > > The reservation will be usually for a certain type of memory then the > > task > > > will probably have to also decide from which pool to allocate the > > segments. > > > I would suggest we create a memory manager per slot and give it memory > > > limit of the slot then we do not have this kind of mixed operation > > > because Dataset/Batch jobs need only segment memory allocations and > > > streaming jobs need only memory chunks for state backends as I > understand > > > the current plan. > > > I would suggest we will look at it if we have the mixed operations at > > some > > > point and it becomes a problem. > > > > > > Thanks, > > > Andrey > > > > > > On Fri, Sep 13, 2019 at 5:24 PM Andrey Zagrebin <[hidden email]> > > > wrote: > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > From: Xintong Song <[hidden email]> > > > > Date: Thu, Sep 12, 2019 at 4:21 AM > > > > Subject: Re: [DISCUSS] FLIP-49: Unified Memory Configuration for > > > > TaskExecutors > > > > To: dev <[hidden email]> > > > > > > > > > > > > Hi Andrey, > > > > > > > > Thanks for bringing this up. > > > > > > > > If I understand correctly, this issue only occurs where the cluster > is > > > > configured with both on-heap and off-heap memory. There should be no > > > > regression for clusters configured in the old way (either all on-heap > > or > > > > all off-heap). > > > > > > > > I also agree that it would be good if the DataSet API jobs can use > both > > > > memory types. The only question I can see is that, from which pool > > (heap > > > / > > > > off-heap) should we allocate memory for DataSet API operators? Do we > > > > always prioritize one pool over the other? Or do we always prioritize > > the > > > > pool with more available memory left? > > > > > > > > Thank you~ > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 8:15 PM Andrey Zagrebin < > [hidden email]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > While looking more into the implementation details of Step 4, we > > > released > > > > > during some offline discussions with @Till > > > > > that there can be a performance degradation for the batch DataSet > API > > > if > > > > we > > > > > simply continue to pull memory from the pool > > > > > according the legacy option taskmanager.memory.off-heap. > > > > > > > > > > The reason is that if the cluster is newly configured to statically > > > split > > > > > heap/off-heap (not like previously either heap or 0ff-heap) > > > > > then the batch DataSet API jobs will be able to use only one type > of > > > > > memory. Although it does not really matter where the memory > segments > > > come > > > > > from > > > > > and potentially batch jobs can use both. Also, currently the > Dataset > > > API > > > > > does not result in absolute resource requirements and its batch > jobs > > > will > > > > > always get a default share of TM resources. > > > > > > > > > > The suggestion is that we let the batch tasks of Dataset API pull > > from > > > > both > > > > > pools according to their fair slot share of each memory type. > > > > > For that we can have a special wrapping view of both pools which > will > > > > pull > > > > > segments (can be randomly) according to the slot limits. > > > > > The view can wrap TM level memory pools and be given to the Task. > > > > > > > > > > Best, > > > > > Andrey > > > > > > > > > > On Mon, Sep 2, 2019 at 1:35 PM Xintong Song <[hidden email] > > > > > > wrote: > > > > > > > > > > > Thanks for your comments, Andrey. > > > > > > > > > > > > - Regarding Task Off-Heap Memory, I think you're right that the > > user > > > > need > > > > > > to make sure that direct memory and native memory together used > by > > > the > > > > > user > > > > > > code (external libs) do not exceed the configured value. As far > as > > I > > > > can > > > > > > think of, there is nothing we can do about it. > > > > > > > > > > > > I addressed the rest of your comment in the wiki page [1]. Please > > > take > > > > a > > > > > > look. > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > On Mon, Sep 2, 2019 at 6:13 PM Andrey Zagrebin < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > EDIT: sorry for confusion I meant > > > > > > > taskmanager.memory.off-heap > > > > > > > instead of > > > > > > > setting taskmanager.memory.preallocate > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 11:29 AM Andrey Zagrebin < > > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > @Xitong thanks a lot for driving the discussion. > > > > > > > > > > > > > > > > I also reviewed the FLIP and it looks quite good to me. > > > > > > > > Here are some comments: > > > > > > > > > > > > > > > > > > > > > > > > - One thing I wanted to discuss is the > > backwards-compatibility > > > > > with > > > > > > > > the previous user setups. We could list which options we > > plan > > > to > > > > > > > deprecate. > > > > > > > > From the first glance it looks possible to provide the > > > > > same/similar > > > > > > > > behaviour for the setups relying on the deprecated > options. > > > E.g. > > > > > > > > setting taskmanager.memory.preallocate to true could > > override > > > > the > > > > > > > > new taskmanager.memory.managed.offheap-fraction to 1 etc. > At > > > the > > > > > > > moment the > > > > > > > > FLIP just states that in some cases it may require > > > > re-configuring > > > > > of > > > > > > > > cluster if migrated from prior versions. My suggestion is > > that > > > > we > > > > > > try > > > > > > > to > > > > > > > > keep it backwards-compatible unless there is a good reason > > > like > > > > > some > > > > > > > major > > > > > > > > complication for the implementation. > > > > > > > > > > > > > > > > > > > > > > > > Also couple of smaller things: > > > > > > > > > > > > > > > > - I suggest we remove TaskExecutorSpecifics from the FLIP > > and > > > > > leave > > > > > > > > some general wording atm, like 'data structure to store' > or > > > > > 'utility > > > > > > > > classes'. When the classes are implemented, we put the > > > concrete > > > > > > class > > > > > > > > names. This way we can avoid confusion and stale > documents. > > > > > > > > > > > > > > > > > > > > > > > > - As I understand, if user task uses native memory (not > > direct > > > > > > memory, > > > > > > > > but e.g. unsafe.allocate or from external lib), there will > > be > > > no > > > > > > > > explicit guard against exceeding 'task off heap memory'. > > Then > > > > user > > > > > > > should > > > > > > > > still explicitly make sure that her/his direct buffer > > > allocation > > > > > > plus > > > > > > > any > > > > > > > > other memory usages does not exceed value announced as > 'task > > > off > > > > > > > heap'. I > > > > > > > > guess there is no so much that can be done about it except > > > > > > mentioning > > > > > > > in > > > > > > > > docs, similar to controlling the heap state backend. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrey > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 10:07 AM Yang Wang < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> I also agree that all the configuration should be calculated > > out > > > > of > > > > > > > >> TaskManager. > > > > > > > >> > > > > > > > >> So a full configuration should be generated before > TaskManager > > > > > > started. > > > > > > > >> > > > > > > > >> Override the calculated configurations through -D now seems > > > > better. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> > > > > > > > >> Yang > > > > > > > >> > > > > > > > >> Xintong Song <[hidden email]> 于2019年9月2日周一 > 上午11:39写道： > > > > > > > >> > > > > > > > >> > I just updated the FLIP wiki page [1], with the following > > > > changes: > > > > > > > >> > > > > > > > > >> > - Network memory uses JVM direct memory, and is > accounted > > > > when > > > > > > > >> setting > > > > > > > >> > JVM max direct memory size parameter. > > > > > > > >> > - Use dynamic configurations (`-Dkey=value`) to pass > > > > calculated > > > > > > > >> memory > > > > > > > >> > configs into TaskExecutors, instead of ENV variables. > > > > > > > >> > - Remove 'supporting memory reservation' from the scope > > of > > > > this > > > > > > > FLIP. > > > > > > > >> > > > > > > > > >> > @till @stephan, please take another look see if there are > > any > > > > > other > > > > > > > >> > concerns. > > > > > > > >> > > > > > > > > >> > Thank you~ > > > > > > > >> > > > > > > > > >> > Xintong Song > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > [1] > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > >> > > > > > > > > >> > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song < > > > > > [hidden email] > > > > > > > > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > >> > > Sorry for the late response. > > > > > > > >> > > > > > > > > > >> > > - Regarding the `TaskExecutorSpecifics` naming, let's > > > discuss > > > > > the > > > > > > > >> detail > > > > > > > >> > > in PR. > > > > > > > >> > > - Regarding passing parameters into the `TaskExecutor`, > +1 > > > for > > > > > > using > > > > > > > >> > > dynamic configuration at the moment, given that there > are > > > more > > > > > > > >> questions > > > > > > > >> > to > > > > > > > >> > > be discussed to have a general framework for overwriting > > > > > > > >> configurations > > > > > > > >> > > with ENV variables. > > > > > > > >> > > - Regarding memory reservation, I double checked with Yu > > and > > > > he > > > > > > will > > > > > > > >> take > > > > > > > >> > > care of it. > > > > > > > >> > > > > > > > > > >> > > Thank you~ > > > > > > > >> > > > > > > > > > >> > > Xintong Song > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann < > > > > > > [hidden email] > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > >> > >> What I forgot to add is that we could tackle specifying > > the > > > > > > > >> > configuration > > > > > > > >> > >> fully in an incremental way and that the full > > specification > > > > > > should > > > > > > > be > > > > > > > >> > the > > > > > > > >> > >> desired end state. > > > > > > > >> > >> > > > > > > > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann < > > > > > > > [hidden email]> > > > > > > > >> > >> wrote: > > > > > > > >> > >> > > > > > > > >> > >> > I think our goal should be that the configuration is > > > fully > > > > > > > >> specified > > > > > > > >> > >> when > > > > > > > >> > >> > the process is started. By considering the internal > > > > > calculation > > > > > > > >> step > > > > > > > >> > to > > > > > > > >> > >> be > > > > > > > >> > >> > rather validate existing values and calculate missing > > > ones, > > > > > > these > > > > > > > >> two > > > > > > > >> > >> > proposal shouldn't even conflict (given determinism). > > > > > > > >> > >> > > > > > > > > >> > >> > Since we don't want to change an existing > > > flink-conf.yaml, > > > > > > > >> specifying > > > > > > > >> > >> the > > > > > > > >> > >> > full configuration would require to pass in the > options > > > > > > > >> differently. > > > > > > > >> > >> > > > > > > > > >> > >> > One way could be the ENV variables approach. The > reason > > > why > > > > > I'm > > > > > > > >> trying > > > > > > > >> > >> to > > > > > > > >> > >> > exclude this feature from the FLIP is that I believe > it > > > > > needs a > > > > > > > bit > > > > > > > >> > more > > > > > > > >> > >> > discussion. Just some questions which come to my > mind: > > > What > > > > > > would > > > > > > > >> be > > > > > > > >> > the > > > > > > > >> > >> > exact format (FLINK_KEY_NAME)? Would we support a dot > > > > > separator > > > > > > > >> which > > > > > > > >> > is > > > > > > > >> > >> > supported by some systems (FLINK.KEY.NAME)? If we > > accept > > > > the > > > > > > dot > > > > > > > >> > >> > separator what would be the order of precedence if > > there > > > > are > > > > > > two > > > > > > > >> ENV > > > > > > > >> > >> > variables defined (FLINK_KEY_NAME and FLINK.KEY.NAME > )? > > > > What > > > > > is > > > > > > > the > > > > > > > >> > >> > precedence of env variable vs. dynamic configuration > > > value > > > > > > > >> specified > > > > > > > >> > >> via -D? > > > > > > > >> > >> > > > > > > > > >> > >> > Another approach could be to pass in the dynamic > > > > > configuration > > > > > > > >> values > > > > > > > >> > >> via > > > > > > > >> > >> > `-Dkey=value` to the Flink process. For that we don't > > > have > > > > to > > > > > > > >> change > > > > > > > >> > >> > anything because the functionality already exists. > > > > > > > >> > >> > > > > > > > > >> > >> > Cheers, > > > > > > > >> > >> > Till > > > > > > > >> > >> > > > > > > > > >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan Ewen < > > > > > > [hidden email]> > > > > > > > >> > wrote: > > > > > > > >> > >> > > > > > > > > >> > >> >> I see. Under the assumption of strict determinism > that > > > > > should > > > > > > > >> work. > > > > > > > >> > >> >> > > > > > > > >> > >> >> The original proposal had this point "don't compute > > > inside > > > > > the > > > > > > > TM, > > > > > > > >> > >> compute > > > > > > > >> > >> >> outside and supply a full config", because that > > sounded > > > > more > > > > > > > >> > intuitive. > > > > > > > >> > >> >> > > > > > > > >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann < > > > > > > > >> [hidden email] > > > > > > > >> > > > > > > > > > >> > >> >> wrote: > > > > > > > >> > >> >> > > > > > > > >> > >> >> > My understanding was that before starting the > Flink > > > > > process > > > > > > we > > > > > > > >> > call a > > > > > > > >> > >> >> > utility which calculates these values. I assume > that > > > > this > > > > > > > >> utility > > > > > > > >> > >> will > > > > > > > >> > >> >> do > > > > > > > >> > >> >> > the calculation based on a set of configured > values > > > > > (process > > > > > > > >> > memory, > > > > > > > >> > >> >> flink > > > > > > > >> > >> >> > memory, network memory etc.). Assuming that these > > > values > > > > > > don't > > > > > > > >> > differ > > > > > > > >> > >> >> from > > > > > > > >> > >> >> > the values with which the JVM is started, it > should > > be > > > > > > > possible > > > > > > > >> to > > > > > > > >> > >> >> > recompute them in the Flink process in order to > set > > > the > > > > > > > values. > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen < > > > > > > > [hidden email] > > > > > > > >> > > > > > > > > >> > >> wrote: > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > When computing the values in the JVM process > after > > > it > > > > > > > started, > > > > > > > >> > how > > > > > > > >> > >> >> would > > > > > > > >> > >> >> > > you deal with values like Max Direct Memory, > > > Metaspace > > > > > > size. > > > > > > > >> > native > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > reservation (reduce heap size), etc? All the > > values > > > > that > > > > > > are > > > > > > > >> > >> >> parameters > > > > > > > >> > >> >> > to > > > > > > > >> > >> >> > > the JVM process and that need to be supplied at > > > > process > > > > > > > >> startup? > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann < > > > > > > > >> > >> [hidden email]> > > > > > > > >> > >> >> > > wrote: > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > Thanks for the clarification. I have some more > > > > > comments: > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > - I would actually split the logic to compute > > the > > > > > > process > > > > > > > >> > memory > > > > > > > >> > >> >> > > > requirements and storing the values into two > > > things. > > > > > > E.g. > > > > > > > >> one > > > > > > > >> > >> could > > > > > > > >> > >> >> > name > > > > > > > >> > >> >> > > > the former TaskExecutorProcessUtility and the > > > > latter > > > > > > > >> > >> >> > > > TaskExecutorProcessMemory. But we can discuss > > this > > > > on > > > > > > the > > > > > > > PR > > > > > > > >> > >> since > > > > > > > >> > >> >> it's > > > > > > > >> > >> >> > > > just a naming detail. > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > - Generally, I'm not opposed to making > > > configuration > > > > > > > values > > > > > > > >> > >> >> overridable > > > > > > > >> > >> >> > > by > > > > > > > >> > >> >> > > > ENV variables. I think this is a very good > idea > > > and > > > > > > makes > > > > > > > >> the > > > > > > > >> > >> >> > > > configurability of Flink processes easier. > > > However, > > > > I > > > > > > > think > > > > > > > >> > that > > > > > > > >> > >> >> adding > > > > > > > >> > >> >> > > > this functionality should not be part of this > > FLIP > > > > > > because > > > > > > > >> it > > > > > > > >> > >> would > > > > > > > >> > >> >> > > simply > > > > > > > >> > >> >> > > > widen the scope unnecessarily. > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > The reasons why I believe it is unnecessary > are > > > the > > > > > > > >> following: > > > > > > > >> > >> For > > > > > > > >> > >> >> Yarn > > > > > > > >> > >> >> > > we > > > > > > > >> > >> >> > > > already create write a flink-conf.yaml which > > could > > > > be > > > > > > > >> populated > > > > > > > >> > >> with > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > memory settings. For the other processes it > > should > > > > not > > > > > > > make > > > > > > > >> a > > > > > > > >> > >> >> > difference > > > > > > > >> > >> >> > > > whether the loaded Configuration is populated > > with > > > > the > > > > > > > >> memory > > > > > > > >> > >> >> settings > > > > > > > >> > >> >> > > from > > > > > > > >> > >> >> > > > ENV variables or by using > > > TaskExecutorProcessUtility > > > > > to > > > > > > > >> compute > > > > > > > >> > >> the > > > > > > > >> > >> >> > > missing > > > > > > > >> > >> >> > > > values from the loaded configuration. If the > > > latter > > > > > > would > > > > > > > >> not > > > > > > > >> > be > > > > > > > >> > >> >> > possible > > > > > > > >> > >> >> > > > (wrong or missing configuration values), then > we > > > > > should > > > > > > > not > > > > > > > >> > have > > > > > > > >> > >> >> been > > > > > > > >> > >> >> > > able > > > > > > > >> > >> >> > > > to actually start the process in the first > > place. > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > - Concerning the memory reservation: I agree > > with > > > > you > > > > > > that > > > > > > > >> we > > > > > > > >> > >> need > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > memory reservation functionality to make > > streaming > > > > > jobs > > > > > > > work > > > > > > > >> > with > > > > > > > >> > >> >> > > "managed" > > > > > > > >> > >> >> > > > memory. However, w/o this functionality the > > whole > > > > Flip > > > > > > > would > > > > > > > >> > >> already > > > > > > > >> > >> >> > > bring > > > > > > > >> > >> >> > > > a good amount of improvements to our users > when > > > > > running > > > > > > > >> batch > > > > > > > >> > >> jobs. > > > > > > > >> > >> >> > > > Moreover, by keeping the scope smaller we can > > > > complete > > > > > > the > > > > > > > >> FLIP > > > > > > > >> > >> >> faster. > > > > > > > >> > >> >> > > > Hence, I would propose to address the memory > > > > > reservation > > > > > > > >> > >> >> functionality > > > > > > > >> > >> >> > > as a > > > > > > > >> > >> >> > > > follow up FLIP (which Yu is working on if I'm > > not > > > > > > > mistaken). > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > Cheers, > > > > > > > >> > >> >> > > > Till > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang Wang < > > > > > > > >> > >> [hidden email]> > > > > > > > >> > >> >> > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > Just add my 2 cents. > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > Using environment variables to override the > > > > > > > configuration > > > > > > > >> for > > > > > > > >> > >> >> > different > > > > > > > >> > >> >> > > > > taskmanagers is better. > > > > > > > >> > >> >> > > > > We do not need to generate dedicated > > > > flink-conf.yaml > > > > > > for > > > > > > > >> all > > > > > > > >> > >> >> > > > taskmanagers. > > > > > > > >> > >> >> > > > > A common flink-conf.yam and different > > > environment > > > > > > > >> variables > > > > > > > >> > are > > > > > > > >> > >> >> > enough. > > > > > > > >> > >> >> > > > > By reducing the distributed cached files, it > > > could > > > > > > make > > > > > > > >> > >> launching > > > > > > > >> > >> >> a > > > > > > > >> > >> >> > > > > taskmanager faster. > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > Stephan gives a good suggestion that we > could > > > move > > > > > the > > > > > > > >> logic > > > > > > > >> > >> into > > > > > > > >> > >> >> > > > > "GlobalConfiguration.loadConfig()" method. > > > > > > > >> > >> >> > > > > Maybe the client could also benefit from > this. > > > > > > Different > > > > > > > >> > users > > > > > > > >> > >> do > > > > > > > >> > >> >> not > > > > > > > >> > >> >> > > > have > > > > > > > >> > >> >> > > > > to export FLINK_CONF_DIR to update few > config > > > > > options. > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > Best, > > > > > > > >> > >> >> > > > > Yang > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > Stephan Ewen <[hidden email]> > 于2019年8月28日周三 > > > > > > 上午1:21写道： > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > One note on the Environment Variables and > > > > > > > Configuration > > > > > > > >> > >> >> discussion. > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > My understanding is that passed ENV > > variables > > > > are > > > > > > > added > > > > > > > >> to > > > > > > > >> > >> the > > > > > > > >> > >> >> > > > > > configuration in the > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > >> > >> method > > > > > > > >> > >> >> (or > > > > > > > >> > >> >> > > > > > similar). > > > > > > > >> > >> >> > > > > > For all the code inside Flink, it looks > like > > > the > > > > > > data > > > > > > > >> was > > > > > > > >> > in > > > > > > > >> > >> the > > > > > > > >> > >> >> > > config > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > start with, just that the scripts that > > compute > > > > the > > > > > > > >> > variables > > > > > > > >> > >> can > > > > > > > >> > >> >> > pass > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > values to the process without actually > > needing > > > > to > > > > > > > write > > > > > > > >> a > > > > > > > >> > >> file. > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > For example the > > > > "GlobalConfiguration.loadConfig()" > > > > > > > >> method > > > > > > > >> > >> would > > > > > > > >> > >> >> > take > > > > > > > >> > >> >> > > > any > > > > > > > >> > >> >> > > > > > ENV variable prefixed with "flink" and add > > it > > > > as a > > > > > > > >> config > > > > > > > >> > >> key. > > > > > > > >> > >> >> > > > > > "flink_taskmanager_memory_size=2g" would > > > become > > > > > > > >> > >> >> > > > "taskmanager.memory.size: > > > > > > > >> > >> >> > > > > > 2g". > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM Xintong > > Song < > > > > > > > >> > >> >> > [hidden email]> > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > Thanks for the comments, Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > I've also seen your comments on the wiki > > > page, > > > > > but > > > > > > > >> let's > > > > > > > >> > >> keep > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > > > > discussion here. > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding 'TaskExecutorSpecifics', how > > do > > > > you > > > > > > > think > > > > > > > >> > about > > > > > > > >> > >> >> > naming > > > > > > > >> > >> >> > > it > > > > > > > >> > >> >> > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > > >> > >> >> > > > > > > - Regarding passing memory > configurations > > > into > > > > > > task > > > > > > > >> > >> executors, > > > > > > > >> > >> >> > I'm > > > > > > > >> > >> >> > > in > > > > > > > >> > >> >> > > > > > favor > > > > > > > >> > >> >> > > > > > > of do it via environment variables > rather > > > than > > > > > > > >> > >> configurations, > > > > > > > >> > >> >> > with > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > following two reasons. > > > > > > > >> > >> >> > > > > > > - It is easier to keep the memory > > options > > > > once > > > > > > > >> > calculate > > > > > > > >> > >> >> not to > > > > > > > >> > >> >> > > be > > > > > > > >> > >> >> > > > > > > changed with environment variables > rather > > > than > > > > > > > >> > >> configurations. > > > > > > > >> > >> >> > > > > > > - I'm not sure whether we should write > > the > > > > > > > >> > configuration > > > > > > > >> > >> in > > > > > > > >> > >> >> > > startup > > > > > > > >> > >> >> > > > > > > scripts. Writing changes into the > > > > configuration > > > > > > > files > > > > > > > >> > when > > > > > > > >> > >> >> > running > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > startup scripts does not sounds right to > > me. > > > > Or > > > > > we > > > > > > > >> could > > > > > > > >> > >> make > > > > > > > >> > >> >> a > > > > > > > >> > >> >> > > copy > > > > > > > >> > >> >> > > > of > > > > > > > >> > >> >> > > > > > > configuration files per flink cluster, > and > > > > make > > > > > > the > > > > > > > >> task > > > > > > > >> > >> >> executor > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > load > > > > > > > >> > >> >> > > > > > > from the copy, and clean up the copy > after > > > the > > > > > > > >> cluster is > > > > > > > >> > >> >> > shutdown, > > > > > > > >> > >> >> > > > > which > > > > > > > >> > >> >> > > > > > > is complicated. (I think this is also > what > > > > > Stephan > > > > > > > >> means > > > > > > > >> > in > > > > > > > >> > >> >> his > > > > > > > >> > >> >> > > > comment > > > > > > > >> > >> >> > > > > > on > > > > > > > >> > >> >> > > > > > > the wiki page?) > > > > > > > >> > >> >> > > > > > > - Regarding reserving memory, I think > this > > > > > change > > > > > > > >> should > > > > > > > >> > be > > > > > > > >> > >> >> > > included > > > > > > > >> > >> >> > > > in > > > > > > > >> > >> >> > > > > > > this FLIP. I think a big part of > > motivations > > > > of > > > > > > this > > > > > > > >> FLIP > > > > > > > >> > >> is > > > > > > > >> > >> >> to > > > > > > > >> > >> >> > > unify > > > > > > > >> > >> >> > > > > > > memory configuration for streaming / > batch > > > and > > > > > > make > > > > > > > it > > > > > > > >> > easy > > > > > > > >> > >> >> for > > > > > > > >> > >> >> > > > > > configuring > > > > > > > >> > >> >> > > > > > > rocksdb memory. If we don't support > memory > > > > > > > >> reservation, > > > > > > > >> > >> then > > > > > > > >> > >> >> > > > streaming > > > > > > > >> > >> >> > > > > > jobs > > > > > > > >> > >> >> > > > > > > cannot use managed memory (neither > on-heap > > > or > > > > > > > >> off-heap), > > > > > > > >> > >> which > > > > > > > >> > >> >> > > makes > > > > > > > >> > >> >> > > > > this > > > > > > > >> > >> >> > > > > > > FLIP incomplete. > > > > > > > >> > >> >> > > > > > > - Regarding network memory, I think you > > are > > > > > > right. I > > > > > > > >> > think > > > > > > > >> > >> we > > > > > > > >> > >> >> > > > probably > > > > > > > >> > >> >> > > > > > > don't need to change network stack from > > > using > > > > > > direct > > > > > > > >> > >> memory to > > > > > > > >> > >> >> > > using > > > > > > > >> > >> >> > > > > > unsafe > > > > > > > >> > >> >> > > > > > > native memory. Network memory size is > > > > > > deterministic, > > > > > > > >> > >> cannot be > > > > > > > >> > >> >> > > > reserved > > > > > > > >> > >> >> > > > > > as > > > > > > > >> > >> >> > > > > > > managed memory does, and cannot be > > > overused. I > > > > > > think > > > > > > > >> it > > > > > > > >> > >> also > > > > > > > >> > >> >> > works > > > > > > > >> > >> >> > > if > > > > > > > >> > >> >> > > > > we > > > > > > > >> > >> >> > > > > > > simply keep using direct memory for > > network > > > > and > > > > > > > >> include > > > > > > > >> > it > > > > > > > >> > >> in > > > > > > > >> > >> >> jvm > > > > > > > >> > >> >> > > max > > > > > > > >> > >> >> > > > > > > direct memory size. > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till > > > Rohrmann > > > > < > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > >> > >> >> > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > Hi Xintong, > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > thanks for addressing the comments and > > > > adding > > > > > a > > > > > > > more > > > > > > > >> > >> >> detailed > > > > > > > >> > >> >> > > > > > > > implementation plan. I have a couple > of > > > > > comments > > > > > > > >> > >> concerning > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > > > > > implementation plan: > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > - The name `TaskExecutorSpecifics` is > > not > > > > > really > > > > > > > >> > >> >> descriptive. > > > > > > > >> > >> >> > > > > Choosing > > > > > > > >> > >> >> > > > > > a > > > > > > > >> > >> >> > > > > > > > different name could help here. > > > > > > > >> > >> >> > > > > > > > - I'm not sure whether I would pass > the > > > > memory > > > > > > > >> > >> >> configuration to > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > > > TaskExecutor via environment > variables. > > I > > > > > think > > > > > > it > > > > > > > >> > would > > > > > > > >> > >> be > > > > > > > >> > >> >> > > better > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > write > > > > > > > >> > >> >> > > > > > > > it into the configuration one uses to > > > start > > > > > the > > > > > > TM > > > > > > > >> > >> process. > > > > > > > >> > >> >> > > > > > > > - If possible, I would exclude the > > memory > > > > > > > >> reservation > > > > > > > >> > >> from > > > > > > > >> > >> >> this > > > > > > > >> > >> >> > > > FLIP > > > > > > > >> > >> >> > > > > > and > > > > > > > >> > >> >> > > > > > > > add this as part of a dedicated FLIP. > > > > > > > >> > >> >> > > > > > > > - If possible, then I would exclude > > > changes > > > > to > > > > > > the > > > > > > > >> > >> network > > > > > > > >> > >> >> > stack > > > > > > > >> > >> >> > > > from > > > > > > > >> > >> >> > > > > > > this > > > > > > > >> > >> >> > > > > > > > FLIP. Maybe we can simply say that the > > > > direct > > > > > > > memory > > > > > > > >> > >> needed > > > > > > > >> > >> >> by > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > > network > > > > > > > >> > >> >> > > > > > > > stack is the framework direct memory > > > > > > requirement. > > > > > > > >> > >> Changing > > > > > > > >> > >> >> how > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > is allocated can happen in a second > > step. > > > > This > > > > > > > would > > > > > > > >> > keep > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > scope > > > > > > > >> > >> >> > > > > of > > > > > > > >> > >> >> > > > > > > this > > > > > > > >> > >> >> > > > > > > > FLIP smaller. > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM > Xintong > > > > Song < > > > > > > > >> > >> >> > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Hi everyone, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > I just updated the FLIP document on > > wiki > > > > > [1], > > > > > > > with > > > > > > > >> > the > > > > > > > >> > >> >> > > following > > > > > > > >> > >> >> > > > > > > changes. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > - Removed open question regarding > > > > > > > MemorySegment > > > > > > > >> > >> >> > allocation. > > > > > > > >> > >> >> > > As > > > > > > > >> > >> >> > > > > > > > > discussed, we exclude this topic > > from > > > > the > > > > > > > >> scope of > > > > > > > >> > >> this > > > > > > > >> > >> >> > > FLIP. > > > > > > > >> > >> >> > > > > > > > > - Updated content about JVM > direct > > > > memory > > > > > > > >> > parameter > > > > > > > >> > >> >> > > according > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > recent > > > > > > > >> > >> >> > > > > > > > > discussions, and moved the other > > > > options > > > > > to > > > > > > > >> > >> "Rejected > > > > > > > >> > >> >> > > > > > Alternatives" > > > > > > > >> > >> >> > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > moment. > > > > > > > >> > >> >> > > > > > > > > - Added implementation steps. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > [1] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 PM > > Stephan > > > > > Ewen < > > > > > > > >> > >> >> > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong: Concerning "wait for > > memory > > > > > users > > > > > > > >> before > > > > > > > >> > >> task > > > > > > > >> > >> >> > > dispose > > > > > > > >> > >> >> > > > > and > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > release": I agree, that's how it > > > should > > > > > be. > > > > > > > >> Let's > > > > > > > >> > >> try it > > > > > > > >> > >> >> > out. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong @Jingsong: Concerning " > JVM > > > > does > > > > > > not > > > > > > > >> wait > > > > > > > >> > >> for > > > > > > > >> > >> >> GC > > > > > > > >> > >> >> > > when > > > > > > > >> > >> >> > > > > > > > allocating > > > > > > > >> > >> >> > > > > > > > > > direct memory buffer": There seems > > to > > > be > > > > > > > pretty > > > > > > > >> > >> >> elaborate > > > > > > > >> > >> >> > > logic > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > free > > > > > > > >> > >> >> > > > > > > > > > buffers when allocating new ones. > > See > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Till: Maybe. If we assume that > the > > > JVM > > > > > > > default > > > > > > > >> > works > > > > > > > >> > >> >> (like > > > > > > > >> > >> >> > > > going > > > > > > > >> > >> >> > > > > > > with > > > > > > > >> > >> >> > > > > > > > > > option 2 and not setting > > > > > > > >> "-XX:MaxDirectMemorySize" > > > > > > > >> > at > > > > > > > >> > >> >> all), > > > > > > > >> > >> >> > > > then > > > > > > > >> > >> >> > > > > I > > > > > > > >> > >> >> > > > > > > > think > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > >> > >> >> > > > > > > > > > should be okay to set > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > >> to > > > > > > > >> > >> >> > > > > > > > > > "off_heap_managed_memory + > > > > direct_memory" > > > > > > even > > > > > > > >> if > > > > > > > >> > we > > > > > > > >> > >> use > > > > > > > >> > >> >> > > > RocksDB. > > > > > > > >> > >> >> > > > > > > That > > > > > > > >> > >> >> > > > > > > > > is a > > > > > > > >> > >> >> > > > > > > > > > big if, though, I honestly have no > > > idea > > > > :D > > > > > > > >> Would be > > > > > > > >> > >> >> good to > > > > > > > >> > >> >> > > > > > > understand > > > > > > > >> > >> >> > > > > > > > > > this, though, because this would > > > affect > > > > > > option > > > > > > > >> (2) > > > > > > > >> > >> and > > > > > > > >> > >> >> > option > > > > > > > >> > >> >> > > > > > (1.2). > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 PM > > > Xintong > > > > > > Song < > > > > > > > >> > >> >> > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thanks for the inputs, Jingsong. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Let me try to summarize your > > points. > > > > > > Please > > > > > > > >> > correct > > > > > > > >> > >> >> me if > > > > > > > >> > >> >> > > I'm > > > > > > > >> > >> >> > > > > > > wrong. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - Memory consumers should > > always > > > > > avoid > > > > > > > >> > returning > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > > > > segments > > > > > > > >> > >> >> > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > memory manager while there > are > > > > still > > > > > > > >> > un-cleaned > > > > > > > >> > >> >> > > > structures / > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > may use the memory. > Otherwise, > > it > > > > > would > > > > > > > >> cause > > > > > > > >> > >> >> serious > > > > > > > >> > >> >> > > > > problems > > > > > > > >> > >> >> > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > >> > >> >> > > > > > > > > > > multiple consumers trying to > > use > > > > the > > > > > > same > > > > > > > >> > memory > > > > > > > >> > >> >> > > segment. > > > > > > > >> > >> >> > > > > > > > > > > - JVM does not wait for GC > when > > > > > > > allocating > > > > > > > >> > >> direct > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > > > > buffer. > > > > > > > >> > >> >> > > > > > > > > > > Therefore even we set proper > > max > > > > > direct > > > > > > > >> memory > > > > > > > >> > >> size > > > > > > > >> > >> >> > > limit, > > > > > > > >> > >> >> > > > > we > > > > > > > >> > >> >> > > > > > > may > > > > > > > >> > >> >> > > > > > > > > > still > > > > > > > >> > >> >> > > > > > > > > > > encounter direct memory oom > if > > > the > > > > GC > > > > > > > >> cleaning > > > > > > > >> > >> >> memory > > > > > > > >> > >> >> > > > slower > > > > > > > >> > >> >> > > > > > > than > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > direct memory allocation. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Am I understanding this > correctly? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at 4:21 PM > > > > > > JingsongLee > > > > > > > < > > > > > > > >> > >> >> > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > .invalid> > > > > > > > >> > >> >> > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Hi stephan: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About option 2: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > if additional threads not > > cleanly > > > > shut > > > > > > > down > > > > > > > >> > >> before > > > > > > > >> > >> >> we > > > > > > > >> > >> >> > can > > > > > > > >> > >> >> > > > > exit > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > task: > > > > > > > >> > >> >> > > > > > > > > > > > In the current case of memory > > > reuse, > > > > > it > > > > > > > has > > > > > > > >> > >> freed up > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > > memory > > > > > > > >> > >> >> > > > > > > it > > > > > > > >> > >> >> > > > > > > > > > > > uses. If this memory is used > by > > > > other > > > > > > > tasks > > > > > > > >> > and > > > > > > > >> > >> >> > > > asynchronous > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > >> > >> >> > > > > > > > > > > > of exited task may still be > > > > writing, > > > > > > > there > > > > > > > >> > will > > > > > > > >> > >> be > > > > > > > >> > >> >> > > > > concurrent > > > > > > > >> > >> >> > > > > > > > > security > > > > > > > >> > >> >> > > > > > > > > > > > problems, and even lead to > > errors > > > > in > > > > > > user > > > > > > > >> > >> computing > > > > > > > >> > >> >> > > > results. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > So I think this is a serious > and > > > > > > > intolerable > > > > > > > >> > >> bug, No > > > > > > > >> > >> >> > > matter > > > > > > > >> > >> >> > > > > > what > > > > > > > >> > >> >> > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > option is, it should be > > avoided. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About direct memory cleaned by > > GC: > > > > > > > >> > >> >> > > > > > > > > > > > I don't think it is a good > idea, > > > > I've > > > > > > > >> > >> encountered so > > > > > > > >> > >> >> > many > > > > > > > >> > >> >> > > > > > > > situations > > > > > > > >> > >> >> > > > > > > > > > > > that it's too late for GC to > > > cause > > > > > > > >> > DirectMemory > > > > > > > >> > >> >> OOM. > > > > > > > >> > >> >> > > > Release > > > > > > > >> > >> >> > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > allocate DirectMemory depend > on > > > the > > > > > > type > > > > > > > of > > > > > > > >> > user > > > > > > > >> > >> >> job, > > > > > > > >> > >> >> > > > which > > > > > > > >> > >> >> > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > often beyond our control. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Best, > > > > > > > >> > >> >> > > > > > > > > > > > Jingsong Lee > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > >> > > > ------------------------------------------------------------------ > > > > > > > >> > >> >> > > > > > > > > > > > From:Stephan Ewen < > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Send Time:2019年8月19日(星期一) > 15:56 > > > > > > > >> > >> >> > > > > > > > > > > > To:dev <[hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] FLIP-49: > > > > Unified > > > > > > > >> Memory > > > > > > > >> > >> >> > > Configuration > > > > > > > >> > >> >> > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > TaskExecutors > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > My main concern with option 2 > > > > > (manually > > > > > > > >> release > > > > > > > >> > >> >> memory) > > > > > > > >> > >> >> > > is > > > > > > > >> > >> >> > > > > that > > > > > > > >> > >> >> > > > > > > > > > segfaults > > > > > > > >> > >> >> > > > > > > > > > > > in the JVM send off all sorts > of > > > > > alarms > > > > > > on > > > > > > > >> user > > > > > > > >> > >> >> ends. > > > > > > > >> > >> >> > So > > > > > > > >> > >> >> > > we > > > > > > > >> > >> >> > > > > > need > > > > > > > >> > >> >> > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > guarantee that this never > > happens. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > The trickyness is in tasks > that > > > uses > > > > > > data > > > > > > > >> > >> >> structures / > > > > > > > >> > >> >> > > > > > algorithms > > > > > > > >> > >> >> > > > > > > > > with > > > > > > > >> > >> >> > > > > > > > > > > > additional threads, like hash > > > table > > > > > > > >> spill/read > > > > > > > >> > >> and > > > > > > > >> > >> >> > > sorting > > > > > > > >> > >> >> > > > > > > threads. > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > >> > >> >> > > > > > > > > > > > to ensure that these cleanly > > shut > > > > down > > > > > > > >> before > > > > > > > >> > we > > > > > > > >> > >> can > > > > > > > >> > >> >> > exit > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > task. > > > > > > > >> > >> >> > > > > > > > > > > > I am not sure that we have > that > > > > > > guaranteed > > > > > > > >> > >> already, > > > > > > > >> > >> >> > > that's > > > > > > > >> > >> >> > > > > why > > > > > > > >> > >> >> > > > > > > > option > > > > > > > >> > >> >> > > > > > > > > > 1.1 > > > > > > > >> > >> >> > > > > > > > > > > > seemed simpler to me. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 at 3:42 > PM > > > > > Xintong > > > > > > > >> Song < > > > > > > > >> > >> >> > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thanks for the comments, > > > Stephan. > > > > > > > >> Summarized > > > > > > > >> > in > > > > > > > >> > >> >> this > > > > > > > >> > >> >> > > way > > > > > > > >> > >> >> > > > > > really > > > > > > > >> > >> >> > > > > > > > > makes > > > > > > > >> > >> >> > > > > > > > > > > > > things easier to understand. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'm in favor of option 2, at > > > least > > > > > for > > > > > > > the > > > > > > > >> > >> >> moment. I > > > > > > > >> > >> >> > > > think > > > > > > > >> > >> >> > > > > it > > > > > > > >> > >> >> > > > > > > is > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > difficult to keep it > segfault > > > safe > > > > > for > > > > > > > >> memory > > > > > > > >> > >> >> > manager, > > > > > > > >> > >> >> > > as > > > > > > > >> > >> >> > > > > > long > > > > > > > >> > >> >> > > > > > > as > > > > > > > >> > >> >> > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > always > > > > > > > >> > >> >> > > > > > > > > > > > > de-allocate the memory > segment > > > > when > > > > > it > > > > > > > is > > > > > > > >> > >> released > > > > > > > >> > >> >> > from > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > consumers. Only if the > memory > > > > > consumer > > > > > > > >> > continue > > > > > > > >> > >> >> using > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > > buffer > > > > > > > >> > >> >> > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > segment after releasing it, > in > > > > which > > > > > > > case > > > > > > > >> we > > > > > > > >> > do > > > > > > > >> > >> >> want > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > job > > > > > > > >> > >> >> > > > > > to > > > > > > > >> > >> >> > > > > > > > > fail > > > > > > > >> > >> >> > > > > > > > > > so > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > detect the memory leak > early. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.2, I don't > think > > > this > > > > > is > > > > > > a > > > > > > > >> good > > > > > > > >> > >> idea. > > > > > > > >> > >> >> > Not > > > > > > > >> > >> >> > > > only > > > > > > > >> > >> >> > > > > > > > because > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > assumption (regular GC is > > enough > > > > to > > > > > > > clean > > > > > > > >> > >> direct > > > > > > > >> > >> >> > > buffers) > > > > > > > >> > >> >> > > > > may > > > > > > > >> > >> >> > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > always > > > > > > > >> > >> >> > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > true, but also it makes > harder > > > for > > > > > > > finding > > > > > > > >> > >> >> problems > > > > > > > >> > >> >> > in > > > > > > > >> > >> >> > > > > cases > > > > > > > >> > >> >> > > > > > of > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > overuse. E.g., user > configured > > > > some > > > > > > > direct > > > > > > > >> > >> memory > > > > > > > >> > >> >> for > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > libraries. > > > > > > > >> > >> >> > > > > > > > > > > > > If the library actually use > > more > > > > > > direct > > > > > > > >> > memory > > > > > > > >> > >> >> then > > > > > > > >> > >> >> > > > > > configured, > > > > > > > >> > >> >> > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > > cannot be cleaned by GC > > because > > > > they > > > > > > are > > > > > > > >> > still > > > > > > > >> > >> in > > > > > > > >> > >> >> > use, > > > > > > > >> > >> >> > > > may > > > > > > > >> > >> >> > > > > > lead > > > > > > > >> > >> >> > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > overuse > > > > > > > >> > >> >> > > > > > > > > > > > > of the total container > memory. > > > In > > > > > that > > > > > > > >> case, > > > > > > > >> > >> if it > > > > > > > >> > >> >> > > didn't > > > > > > > >> > >> >> > > > > > touch > > > > > > > >> > >> >> > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > default max direct memory > > limit, > > > > we > > > > > > > cannot > > > > > > > >> > get > > > > > > > >> > >> a > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > OOM > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > it > > > > > > > >> > >> >> > > > > > > > > > > > > will become super hard to > > > > understand > > > > > > > which > > > > > > > >> > >> part of > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > >> > >> >> > > > > > > > > > > > > to be updated. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.1, it has the > > > similar > > > > > > > >> problem as > > > > > > > >> > >> >> 1.2, if > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > > > exceeded > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > memory does not reach the > max > > > > direct > > > > > > > >> memory > > > > > > > >> > >> limit > > > > > > > >> > >> >> > > > specified > > > > > > > >> > >> >> > > > > > by > > > > > > > >> > >> >> > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > dedicated parameter. I think > > it > > > is > > > > > > > >> slightly > > > > > > > >> > >> better > > > > > > > >> > >> >> > than > > > > > > > >> > >> >> > > > > 1.2, > > > > > > > >> > >> >> > > > > > > only > > > > > > > >> > >> >> > > > > > > > > > > because > > > > > > > >> > >> >> > > > > > > > > > > > > we can tune the parameter. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 at 2:53 > > PM > > > > > > Stephan > > > > > > > >> Ewen > > > > > > > >> > < > > > > > > > >> > >> >> > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > About the > > > > > "-XX:MaxDirectMemorySize" > > > > > > > >> > >> discussion, > > > > > > > >> > >> >> > maybe > > > > > > > >> > >> >> > > > let > > > > > > > >> > >> >> > > > > > me > > > > > > > >> > >> >> > > > > > > > > > > summarize > > > > > > > >> > >> >> > > > > > > > > > > > > it a > > > > > > > >> > >> >> > > > > > > > > > > > > > bit differently: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > We have the following two > > > > options: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (1) We let MemorySegments > be > > > > > > > >> de-allocated > > > > > > > >> > by > > > > > > > >> > >> the > > > > > > > >> > >> >> > GC. > > > > > > > >> > >> >> > > > That > > > > > > > >> > >> >> > > > > > > makes > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > >> > >> >> > > > > > > > > > > > > segfault > > > > > > > >> > >> >> > > > > > > > > > > > > > safe. But then we need a > way > > > to > > > > > > > trigger > > > > > > > >> GC > > > > > > > >> > in > > > > > > > >> > >> >> case > > > > > > > >> > >> >> > > > > > > > de-allocation > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > re-allocation of a bunch > of > > > > > segments > > > > > > > >> > happens > > > > > > > >> > >> >> > quickly, > > > > > > > >> > >> >> > > > > which > > > > > > > >> > >> >> > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > often > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > case during batch > scheduling > > > or > > > > > task > > > > > > > >> > restart. > > > > > > > >> > >> >> > > > > > > > > > > > > > - The > > > > "-XX:MaxDirectMemorySize" > > > > > > > >> (option > > > > > > > >> > >> 1.1) > > > > > > > >> > >> >> is > > > > > > > >> > >> >> > one > > > > > > > >> > >> >> > > > way > > > > > > > >> > >> >> > > > > > to > > > > > > > >> > >> >> > > > > > > do > > > > > > > >> > >> >> > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > - Another way could be > to > > > > have a > > > > > > > >> > dedicated > > > > > > > >> > >> >> > > > bookkeeping > > > > > > > >> > >> >> > > > > in > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > MemoryManager (option > 1.2), > > so > > > > > that > > > > > > > this > > > > > > > >> > is a > > > > > > > >> > >> >> > number > > > > > > > >> > >> >> > > > > > > > independent > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > parameter. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (2) We manually allocate > and > > > > > > > de-allocate > > > > > > > >> > the > > > > > > > >> > >> >> memory > > > > > > > >> > >> >> > > for > > > > > > > >> > >> >> > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > MemorySegments > > > > > > > >> > >> >> > > > > > > > > > > > > > (option 2). That way we > need > > > not > > > > > > worry > > > > > > > >> > about > > > > > > > >> > >> >> > > triggering > > > > > > > >> > >> >> > > > > GC > > > > > > > >> > >> >> > > > > > by > > > > > > > >> > >> >> > > > > > > > > some > > > > > > > >> > >> >> > > > > > > > > > > > > > threshold or bookkeeping, > > but > > > it > > > > > is > > > > > > > >> harder > > > > > > > >> > to > > > > > > > >> > >> >> > prevent > > > > > > > >> > >> >> > > > > > > > segfaults. > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > be very careful about when > > we > > > > > > release > > > > > > > >> the > > > > > > > >> > >> memory > > > > > > > >> > >> >> > > > segments > > > > > > > >> > >> >> > > > > > > (only > > > > > > > >> > >> >> > > > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > cleanup phase of the main > > > > thread). > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option 1.1, > we > > > > > > probably > > > > > > > >> need > > > > > > > >> > to > > > > > > > >> > >> >> set > > > > > > > >> > >> >> > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > to > > > > > > > >> > >> >> > > "off_heap_managed_memory + > > > > > > > >> > >> >> > > > > > > > > > > direct_memory" > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > have "direct_memory" as a > > > > separate > > > > > > > >> reserved > > > > > > > >> > >> >> memory > > > > > > > >> > >> >> > > > pool. > > > > > > > >> > >> >> > > > > > > > Because > > > > > > > >> > >> >> > > > > > > > > if > > > > > > > >> > >> >> > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > just > > > > > > > >> > >> >> > > > > > > > > > > > > > set > > "-XX:MaxDirectMemorySize" > > > to > > > > > > > >> > >> >> > > > > "off_heap_managed_memory + > > > > > > > >> > >> >> > > > > > > > > > > > > jvm_overhead", > > > > > > > >> > >> >> > > > > > > > > > > > > > then there will be times > > when > > > > that > > > > > > > >> entire > > > > > > > >> > >> >> memory is > > > > > > > >> > >> >> > > > > > allocated > > > > > > > >> > >> >> > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > buffers and we have > nothing > > > left > > > > > for > > > > > > > the > > > > > > > >> > JVM > > > > > > > >> > >> >> > > overhead. > > > > > > > >> > >> >> > > > So > > > > > > > >> > >> >> > > > > > we > > > > > > > >> > >> >> > > > > > > > > either > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > >> > >> >> > > > > > > > > > > > > a > > > > > > > >> > >> >> > > > > > > > > > > > > > way to compensate for that > > > > (again > > > > > > some > > > > > > > >> > safety > > > > > > > >> > >> >> > margin > > > > > > > >> > >> >> > > > > cutoff > > > > > > > >> > >> >> > > > > > > > > value) > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > will exceed container > > memory. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option 1.2, > we > > > > need > > > > > to > > > > > > > be > > > > > > > >> > aware > > > > > > > >> > >> >> that > > > > > > > >> > >> >> > it > > > > > > > >> > >> >> > > > > takes > > > > > > > >> > >> >> > > > > > > > > > elaborate > > > > > > > >> > >> >> > > > > > > > > > > > > logic > > > > > > > >> > >> >> > > > > > > > > > > > > > to push recycling of > direct > > > > > buffers > > > > > > > >> without > > > > > > > >> > >> >> always > > > > > > > >> > >> >> > > > > > > triggering a > > > > > > > >> > >> >> > > > > > > > > > full > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My first guess is that the > > > > options > > > > > > > will > > > > > > > >> be > > > > > > > >> > >> >> easiest > > > > > > > >> > >> >> > to > > > > > > > >> > >> >> > > > do > > > > > > > >> > >> >> > > > > in > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > following > > > > > > > >> > >> >> > > > > > > > > > > > > > order: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.1 with a > > > dedicated > > > > > > > >> > direct_memory > > > > > > > >> > >> >> > > > parameter, > > > > > > > >> > >> >> > > > > as > > > > > > > >> > >> >> > > > > > > > > > discussed > > > > > > > >> > >> >> > > > > > > > > > > > > > above. We would need to > > find a > > > > way > > > > > > to > > > > > > > >> set > > > > > > > >> > the > > > > > > > >> > >> >> > > > > direct_memory > > > > > > > >> > >> >> > > > > > > > > > parameter > > > > > > > >> > >> >> > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > default. We could start > with > > > 64 > > > > MB > > > > > > and > > > > > > > >> see > > > > > > > >> > >> how > > > > > > > >> > >> >> it > > > > > > > >> > >> >> > > goes > > > > > > > >> > >> >> > > > in > > > > > > > >> > >> >> > > > > > > > > practice. > > > > > > > >> > >> >> > > > > > > > > > > One > > > > > > > >> > >> >> > > > > > > > > > > > > > danger I see is that > setting > > > > this > > > > > > loo > > > > > > > >> low > > > > > > > >> > can > > > > > > > >> > >> >> > cause a > > > > > > > >> > >> >> > > > > bunch > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > additional > > > > > > > >> > >> >> > > > > > > > > > > > > > GCs compared to before (we > > > need > > > > to > > > > > > > watch > > > > > > > >> > this > > > > > > > >> > >> >> > > > carefully). > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 2. It is > actually > > > > quite > > > > > > > >> simple > > > > > > > >> > to > > > > > > > >> > >> >> > > implement, > > > > > > > >> > >> >> > > > > we > > > > > > > >> > >> >> > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > try > > > > > > > >> > >> >> > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > segfault safe we are at > the > > > > > moment. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.2: We would > not > > > > touch > > > > > > the > > > > > > > >> > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > >> > >> >> > > > > > > > > > > > > parameter > > > > > > > >> > >> >> > > > > > > > > > > > > > at all and assume that all > > the > > > > > > direct > > > > > > > >> > memory > > > > > > > >> > >> >> > > > allocations > > > > > > > >> > >> >> > > > > > that > > > > > > > >> > >> >> > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > Netty do are infrequent > > enough > > > > to > > > > > be > > > > > > > >> > cleaned > > > > > > > >> > >> up > > > > > > > >> > >> >> > fast > > > > > > > >> > >> >> > > > > enough > > > > > > > >> > >> >> > > > > > > > > through > > > > > > > >> > >> >> > > > > > > > > > > > > regular > > > > > > > >> > >> >> > > > > > > > > > > > > > GC. I am not sure if that > > is a > > > > > valid > > > > > > > >> > >> assumption, > > > > > > > >> > >> >> > > > though. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Best, > > > > > > > >> > >> >> > > > > > > > > > > > > > Stephan > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at > 2:16 > > > PM > > > > > > > Xintong > > > > > > > >> > Song > > > > > > > >> > >> < > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing your > > > > opinion > > > > > > > Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor of > > > > alternative > > > > > > 2. > > > > > > > I > > > > > > > >> was > > > > > > > >> > >> >> > wondering > > > > > > > >> > >> >> > > > > > whether > > > > > > > >> > >> >> > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > >> > >> >> > > > > > > > > > > > > > > using Unsafe.allocate() > > for > > > > > > off-heap > > > > > > > >> > >> managed > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > > and > > > > > > > >> > >> >> > > > > > > > network > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But after > > > > giving > > > > > > it a > > > > > > > >> > second > > > > > > > >> > >> >> > > thought, > > > > > > > >> > >> >> > > > I > > > > > > > >> > >> >> > > > > > > think > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 using > direct > > > > > memory > > > > > > > for > > > > > > > >> > >> off-heap > > > > > > > >> > >> >> > > > managed > > > > > > > >> > >> >> > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your concern, > I > > > > think > > > > > > what > > > > > > > >> > >> proposed > > > > > > > >> > >> >> in > > > > > > > >> > >> >> > > this > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > >> > >> >> > > > > > > it > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed memory > > and > > > > > > network > > > > > > > >> > memory > > > > > > > >> > >> >> > > allocated > > > > > > > >> > >> >> > > > > > > through > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), which > > > means > > > > > > they > > > > > > > >> are > > > > > > > >> > >> >> > practically > > > > > > > >> > >> >> > > > > > native > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > direct > > > > > memory. > > > > > > > The > > > > > > > >> > only > > > > > > > >> > >> >> parts > > > > > > > >> > >> >> > of > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory are > task > > > > > > off-heap > > > > > > > >> > memory > > > > > > > >> > >> and > > > > > > > >> > >> >> > JVM > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly alternative 2 > > > suggests > > > > > to > > > > > > > set > > > > > > > >> the > > > > > > > >> > >> JVM > > > > > > > >> > >> >> max > > > > > > > >> > >> >> > > > > direct > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at > > 1:48 > > > > PM > > > > > > Till > > > > > > > >> > >> Rohrmann > > > > > > > >> > >> >> < > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > clarification > > > > > > > >> Xintong. I > > > > > > > >> > >> >> > > understand > > > > > > > >> > >> >> > > > > the > > > > > > > >> > >> >> > > > > > > two > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in favour > of > > > > > option 2 > > > > > > > >> > because > > > > > > > >> > >> it > > > > > > > >> > >> >> > makes > > > > > > > >> > >> >> > > > > > things > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the direct > > > > > memory, I > > > > > > > >> fear > > > > > > > >> > >> that > > > > > > > >> > >> >> we > > > > > > > >> > >> >> > > might > > > > > > > >> > >> >> > > > > end > > > > > > > >> > >> >> > > > > > > up > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we are > > > > currently > > > > > > in: > > > > > > > >> The > > > > > > > >> > >> user > > > > > > > >> > >> >> > might > > > > > > > >> > >> >> > > > see > > > > > > > >> > >> >> > > > > > that > > > > > > > >> > >> >> > > > > > > > her > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS and > > does > > > > not > > > > > > know > > > > > > > >> why > > > > > > > >> > >> this > > > > > > > >> > >> >> is > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > case. > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease the > > > > process > > > > > > > memory > > > > > > > >> > size > > > > > > > >> > >> >> > > (similar > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order to > > > > accommodate > > > > > > for > > > > > > > >> the > > > > > > > >> > >> extra > > > > > > > >> > >> >> > > direct > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > memory > > > > > budgets > > > > > > > >> which > > > > > > > >> > >> are > > > > > > > >> > >> >> not > > > > > > > >> > >> >> > > > fully > > > > > > > >> > >> >> > > > > > used > > > > > > > >> > >> >> > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the overall > > memory > > > > > > > >> consumption. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > at > > > > 11:01 > > > > > AM > > > > > > > >> > Xintong > > > > > > > >> > >> >> Song < > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain this > > > with a > > > > > > > >> concrete > > > > > > > >> > >> >> example > > > > > > > >> > >> >> > > Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we have > the > > > > > > following > > > > > > > >> > >> scenario. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > Memory: > > > 1GB > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct Memory > > (Task > > > > > > Off-Heap > > > > > > > >> > >> Memory + > > > > > > > >> > >> >> JVM > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory (JVM > Heap > > > > > Memory, > > > > > > > JVM > > > > > > > >> > >> >> Metaspace, > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network Memory): > 800MB > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 2, > we > > > set > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 3, > we > > > set > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > a > > > > > > > >> > >> >> > > > > > > > very > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual direct > > > > memory > > > > > > > usage > > > > > > > >> of > > > > > > > >> > >> Task > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed 200MB, > > > then > > > > > > > >> > alternative 2 > > > > > > > >> > >> >> and > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > Setting > > > > larger > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > > > > will > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the other > > > memory > > > > > > pools. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual direct > > > > memory > > > > > > > usage > > > > > > > >> of > > > > > > > >> > >> Task > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead potentially > > > > exceed > > > > > > > 200MB, > > > > > > > >> > then > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - Alternative 2 > > > suffers > > > > > > from > > > > > > > >> > >> frequent > > > > > > > >> > >> >> OOM. > > > > > > > >> > >> >> > > To > > > > > > > >> > >> >> > > > > > avoid > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do is to > > > > modify > > > > > > the > > > > > > > >> > >> >> configuration > > > > > > > >> > >> >> > > and > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task Off-Heap > > > Memory + > > > > > JVM > > > > > > > >> > >> Overhead). > > > > > > > >> > >> >> > Let's > > > > > > > >> > >> >> > > > say > > > > > > > >> > >> >> > > > > > > that > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct Memory to > > > 250MB, > > > > > > this > > > > > > > >> will > > > > > > > >> > >> >> reduce > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > > total > > > > > > > >> > >> >> > > > > > > > size > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to 750MB, > > given > > > > the > > > > > > > total > > > > > > > >> > >> process > > > > > > > >> > >> >> > > memory > > > > > > > >> > >> >> > > > > > > remains > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For alternative > > 3, > > > > > there > > > > > > is > > > > > > > >> no > > > > > > > >> > >> >> chance of > > > > > > > >> > >> >> > > > > direct > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of exceeding the > > > total > > > > > > > process > > > > > > > >> > >> memory > > > > > > > >> > >> >> > limit, > > > > > > > >> > >> >> > > > but > > > > > > > >> > >> >> > > > > > > given > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up all > the > > > > > reserved > > > > > > > >> native > > > > > > > >> > >> >> memory > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > Metaspace), > > > > > if > > > > > > > the > > > > > > > >> > >> actual > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very close to > > > > 200MB, > > > > > > user > > > > > > > >> > >> probably > > > > > > > >> > >> >> do > > > > > > > >> > >> >> > > not > > > > > > > >> > >> >> > > > > need > > > > > > > >> > >> >> > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > configurations. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I think > > from > > > > the > > > > > > > user's > > > > > > > >> > >> >> > > perspective, a > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for alternative 2 > may > > > lead > > > > > to > > > > > > > >> lower > > > > > > > >> > >> >> resource > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > > 10:28 > > > > > > AM > > > > > > > >> Till > > > > > > > >> > >> >> > Rohrmann > > > > > > > >> > >> >> > > < > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you have > to > > > help > > > > > me > > > > > > > >> > >> understand > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > > > difference > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > memory > > > > under > > > > > > > >> > utilization > > > > > > > >> > >> >> > > Xintong. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: > set > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > Task > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. Then > there > > > is > > > > > the > > > > > > > risk > > > > > > > >> > that > > > > > > > >> > >> >> this > > > > > > > >> > >> >> > > size > > > > > > > >> > >> >> > > > > is > > > > > > > >> > >> >> > > > > > > too > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of garbage > > > > collection > > > > > > and > > > > > > > >> > >> >> potentially > > > > > > > >> > >> >> > an > > > > > > > >> > >> >> > > > OOM. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 3: > set > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > > > > something > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 2. > This > > > > would > > > > > of > > > > > > > >> course > > > > > > > >> > >> >> reduce > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > sizes > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > alternative > > > 2 > > > > > now > > > > > > > >> result > > > > > > > >> > >> in an > > > > > > > >> > >> >> > > under > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > alternative > > > > 3? > > > > > > If > > > > > > > >> > >> >> alternative 3 > > > > > > > >> > >> >> > > > > > strictly > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct memory size > > and > > > > we > > > > > > use > > > > > > > >> only > > > > > > > >> > >> >> little, > > > > > > > >> > >> >> > > > then I > > > > > > > >> > >> >> > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 3 > > results > > > in > > > > > > > memory > > > > > > > >> > under > > > > > > > >> > >> >> > > > > utilization. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, > 2019 > > > at > > > > > 4:19 > > > > > > > PM > > > > > > > >> > Yang > > > > > > > >> > >> >> Wang < > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi xintong,till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native and > > Direct > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point is > > setting > > > a > > > > > very > > > > > > > >> large > > > > > > > >> > >> max > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > size > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > differentiate > > direct > > > > and > > > > > > > >> native > > > > > > > >> > >> >> memory. > > > > > > > >> > >> >> > If > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct memory > and > > > > > > framework > > > > > > > >> > direct > > > > > > > >> > >> >> > > > memory,could > > > > > > > >> > >> >> > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in favor of > > > > setting > > > > > > > >> direct > > > > > > > >> > >> memory > > > > > > > >> > >> >> > with > > > > > > > >> > >> >> > > > > fixed > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > Calculation > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree with > > > xintong. > > > > > For > > > > > > > Yarn > > > > > > > >> > and > > > > > > > >> > >> >> k8s,we > > > > > > > >> > >> >> > > > need > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > check > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configurations > in > > > > client > > > > > > to > > > > > > > >> avoid > > > > > > > >> > >> >> > > submitting > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > master. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < > > > > > > > >> > >> [hidden email] > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks for > > > replying, > > > > > > Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > MemorySegment, I > > > > > > > think > > > > > > > >> > you > > > > > > > >> > >> are > > > > > > > >> > >> >> > > right > > > > > > > >> > >> >> > > > > that > > > > > > > >> > >> >> > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in the > > scope > > > > of > > > > > > this > > > > > > > >> > FLIP. > > > > > > > >> > >> >> This > > > > > > > >> > >> >> > > FLIP > > > > > > > >> > >> >> > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configure > memory > > > > pools > > > > > > for > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > >> > >> >> > > > > with > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > consumers > > > use > > > > > it. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About direct > > > > memory, I > > > > > > > think > > > > > > > >> > >> >> > alternative > > > > > > > >> > >> >> > > 3 > > > > > > > >> > >> >> > > > > may > > > > > > > >> > >> >> > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > reservation > > issue > > > > that > > > > > > > >> > >> alternative 2 > > > > > > > >> > >> >> > > does, > > > > > > > >> > >> >> > > > > but > > > > > > > >> > >> >> > > > > > at > > > > > > > >> > >> >> > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using memory > at > > > the > > > > > > > >> container > > > > > > > >> > >> level, > > > > > > > >> > >> >> > > which > > > > > > > >> > >> >> > > > is > > > > > > > >> > >> >> > > > > > not > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both "Task > > > Off-Heap > > > > > > > Memory" > > > > > > > >> and > > > > > > > >> > >> "JVM > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > >> > >> >> > > > > > > are > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative 2, > > > users > > > > > > might > > > > > > > >> > >> configure > > > > > > > >> > >> >> > them > > > > > > > >> > >> >> > > > > > higher > > > > > > > >> > >> >> > > > > > > > than > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to avoid > > > > getting > > > > > a > > > > > > > >> direct > > > > > > > >> > >> OOM. > > > > > > > >> > >> >> For > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct OOM, so > > > they > > > > > may > > > > > > > not > > > > > > > >> > >> config > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > two > > > > > > > >> > >> >> > > > > > > options > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > consequences > > > are > > > > > > risks > > > > > > > >> of > > > > > > > >> > >> >> overall > > > > > > > >> > >> >> > > > > container > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug > 13, > > > 2019 > > > > > at > > > > > > > >> 9:39 AM > > > > > > > >> > >> Till > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > >> > >> >> > > > > > > > > > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for > > > > proposing > > > > > > > this > > > > > > > >> > FLIP > > > > > > > >> > >> >> > Xintong. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in all I > > > think > > > > > it > > > > > > > >> already > > > > > > > >> > >> >> looks > > > > > > > >> > >> >> > > quite > > > > > > > >> > >> >> > > > > > good. > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > question > about > > > > > > > allocating > > > > > > > >> > >> memory > > > > > > > >> > >> >> > > > segments, > > > > > > > >> > >> >> > > > > I > > > > > > > >> > >> >> > > > > > > was > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > strictly > > > necessary > > > > > to > > > > > > do > > > > > > > >> in > > > > > > > >> > the > > > > > > > >> > >> >> > context > > > > > > > >> > >> >> > > > of > > > > > > > >> > >> >> > > > > > this > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be done as a > > > > follow > > > > > > up? > > > > > > > >> > Without > > > > > > > >> > >> >> > knowing > > > > > > > >> > >> >> > > > all > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that we > would > > > > widen > > > > > > the > > > > > > > >> scope > > > > > > > >> > >> of > > > > > > > >> > >> >> this > > > > > > > >> > >> >> > > > FLIP > > > > > > > >> > >> >> > > > > > too > > > > > > > >> > >> >> > > > > > > > much > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to touch all > > the > > > > > > > existing > > > > > > > >> > call > > > > > > > >> > >> >> sites > > > > > > > >> > >> >> > of > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > segments > > > > > (this > > > > > > > >> should > > > > > > > >> > >> >> mainly > > > > > > > >> > >> >> > be > > > > > > > >> > >> >> > > > > batch > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the memory > > > > > reservation > > > > > > > >> call > > > > > > > >> > to > > > > > > > >> > >> the > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this and I > > would > > > > > hope > > > > > > > that > > > > > > > >> > >> this is > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > only > > > > > > > >> > >> >> > > > > > > point > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > streaming > job > > > > would > > > > > > have > > > > > > > >> with > > > > > > > >> > >> the > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Concerning > the > > > > > second > > > > > > > open > > > > > > > >> > >> >> question > > > > > > > >> > >> >> > > about > > > > > > > >> > >> >> > > > > > > setting > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > memory > > > > > limit, I > > > > > > > >> would > > > > > > > >> > >> also > > > > > > > >> > >> >> be > > > > > > > >> > >> >> > > > > > interested > > > > > > > >> > >> >> > > > > > > > why > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > leaving it > > open > > > > > would > > > > > > be > > > > > > > >> > best. > > > > > > > >> > >> My > > > > > > > >> > >> >> > > concern > > > > > > > >> > >> >> > > > > > about > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in a > > similar > > > > > > > situation > > > > > > > >> as > > > > > > > >> > we > > > > > > > >> > >> >> are > > > > > > > >> > >> >> > now > > > > > > > >> > >> >> > > > > with > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > different > > > > memory > > > > > > > pools > > > > > > > >> > are > > > > > > > >> > >> not > > > > > > > >> > >> >> > > > clearly > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a different > > > pool, > > > > > then > > > > > > > it > > > > > > > >> is > > > > > > > >> > >> quite > > > > > > > >> > >> >> > hard > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > process to > get > > > > > killed > > > > > > > for > > > > > > > >> > using > > > > > > > >> > >> >> too > > > > > > > >> > >> >> > > much > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead to a > > > similar > > > > > > > >> situation > > > > > > > >> > >> what > > > > > > > >> > >> >> we > > > > > > > >> > >> >> > > have > > > > > > > >> > >> >> > > > > with > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > setting a > sane > > > > > default > > > > > > > >> value > > > > > > > >> > >> for > > > > > > > >> > >> >> max > > > > > > > >> > >> >> > > > direct > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option to > > > increase > > > > > it > > > > > > if > > > > > > > >> he > > > > > > > >> > >> runs > > > > > > > >> > >> >> into > > > > > > > >> > >> >> > > an > > > > > > > >> > >> >> > > > > OOM. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, > how > > > > would > > > > > > > >> > >> alternative 2 > > > > > > > >> > >> >> > lead > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > > lower > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > alternative > 3 > > > > where > > > > > we > > > > > > > set > > > > > > > >> > the > > > > > > > >> > >> >> direct > > > > > > > >> > >> >> > > > > memory > > > > > > > >> > >> >> > > > > > > to a > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, Aug > 9, > > > > 2019 > > > > > at > > > > > > > >> 9:12 > > > > > > > >> > AM > > > > > > > >> > >> >> > Xintong > > > > > > > >> > >> >> > > > > Song < > > > > > > > >> > >> >> > > > > > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks for > > the > > > > > > > feedback, > > > > > > > >> > >> Yang. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Regarding > > your > > > > > > > comments: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Native > and > > > > Direct > > > > > > > >> Memory* > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > > setting > > > > a > > > > > > very > > > > > > > >> > large > > > > > > > >> > >> max > > > > > > > >> > >> >> > > direct > > > > > > > >> > >> >> > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > sides. > > > > E.g., > > > > > we > > > > > > > do > > > > > > > >> not > > > > > > > >> > >> >> worry > > > > > > > >> > >> >> > > about > > > > > > > >> > >> >> > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > allocate > > > > > managed > > > > > > / > > > > > > > >> > network > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > > with > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > However, > > there > > > > are > > > > > > > also > > > > > > > >> > some > > > > > > > >> > >> >> down > > > > > > > >> > >> >> > > sides > > > > > > > >> > >> >> > > > > of > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - One > > > thing I > > > > > can > > > > > > > >> think > > > > > > > >> > >> of is > > > > > > > >> > >> >> > that > > > > > > > >> > >> >> > > > if > > > > > > > >> > >> >> > > > > a > > > > > > > >> > >> >> > > > > > > task > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > killed > > due > > > to > > > > > > > >> overusing > > > > > > > >> > >> >> memory, > > > > > > > >> > >> >> > it > > > > > > > >> > >> >> > > > > could > > > > > > > >> > >> >> > > > > > > be > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > of the > > > memory > > > > > is > > > > > > > >> > overused. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > Another > > > > down > > > > > > side > > > > > > > >> is > > > > > > > >> > >> that > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > JVM > > > > > > > >> > >> >> > > > > > never > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > direct > > > memory > > > > > > > limit, > > > > > > > >> > >> because > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > limit > > > > > > > >> > >> >> > > > > > is > > > > > > > >> > >> >> > > > > > > > too > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > means > we > > > kind > > > > > of > > > > > > > >> relay > > > > > > > >> > on > > > > > > > >> > >> >> heap > > > > > > > >> > >> >> > > > memory > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory. > > > That > > > > > > could > > > > > > > >> be a > > > > > > > >> > >> >> problem > > > > > > > >> > >> >> > in > > > > > > > >> > >> >> > > > > cases > > > > > > > >> > >> >> > > > > > > > where > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > usage > but > > > not > > > > > > > enough > > > > > > > >> > heap > > > > > > > >> > >> >> > activity > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe you > > can > > > > > share > > > > > > > your > > > > > > > >> > >> reasons > > > > > > > >> > >> >> > for > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if there > are > > > > > > anything > > > > > > > >> else > > > > > > > >> > I > > > > > > > >> > >> >> > > > overlooked. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > Calculation* > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If there > is > > > any > > > > > > > conflict > > > > > > > >> > >> between > > > > > > > >> > >> >> > > > multiple > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > specified, > > > > > > > I > > > > > > > >> > >> think we > > > > > > > >> > >> >> > > should > > > > > > > >> > >> >> > > > > > throw > > > > > > > >> > >> >> > > > > > > > an > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > doing > > > > > > checking > > > > > > > >> on > > > > > > > >> > the > > > > > > > >> > >> >> > client > > > > > > > >> > >> >> > > > side > > > > > > > >> > >> >> > > > > > is > > > > > > > >> > >> >> > > > > > > a > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we can > > > > > discover > > > > > > > the > > > > > > > >> > >> problem > > > > > > > >> > >> >> > > before > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is always > a > > > good > > > > > > > thing. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But we can > > not > > > > > only > > > > > > > >> rely on > > > > > > > >> > >> the > > > > > > > >> > >> >> > > client > > > > > > > >> > >> >> > > > > side > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > standalone > > > > cluster > > > > > > > >> > >> TaskManagers > > > > > > > >> > >> >> on > > > > > > > >> > >> >> > > > > > different > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > different > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > and > > > > > > the > > > > > > > >> > client > > > > > > > >> > >> >> does > > > > > > > >> > >> >> > > see > > > > > > > >> > >> >> > > > > > that. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What do > you > > > > think? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong > Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, > Aug > > 8, > > > > > 2019 > > > > > > at > > > > > > > >> 5:09 > > > > > > > >> > >> PM > > > > > > > >> > >> >> Yang > > > > > > > >> > >> >> > > > Wang > > > > > > > >> > >> >> > > > > < > > > > > > > >> > >> >> > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > xintong, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > for > > > > your > > > > > > > >> detailed > > > > > > > >> > >> >> > proposal. > > > > > > > >> > >> >> > > > > After > > > > > > > >> > >> >> > > > > > > all > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > it > > > > > > will > > > > > > > be > > > > > > > >> > more > > > > > > > >> > >> >> > > powerful > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > control > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > just > have > > > few > > > > > > > >> questions > > > > > > > >> > >> about > > > > > > > >> > >> >> it. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > Native > > > > and > > > > > > > Direct > > > > > > > >> > >> Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We do > not > > > > > > > >> differentiate > > > > > > > >> > >> user > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > included > > in > > > > task > > > > > > > >> off-heap > > > > > > > >> > >> >> memory. > > > > > > > >> > >> >> > > > > Right? > > > > > > > >> > >> >> > > > > > > So i > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > properly. I > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > Memory > > > > > > > >> Calculation > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If the > sum > > > of > > > > > and > > > > > > > >> > >> fine-grained > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > larger > > > than > > > > > > total > > > > > > > >> > >> process > > > > > > > >> > >> >> > > memory, > > > > > > > >> > >> >> > > > > how > > > > > > > >> > >> >> > > > > > do > > > > > > > >> > >> >> > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we need > to > > > > check > > > > > > the > > > > > > > >> > memory > > > > > > > >> > >> >> > > > > configuration > > > > > > > >> > >> >> > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong > > > Song < > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > everyone, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > would > > > > like > > > > > to > > > > > > > >> start > > > > > > > >> > a > > > > > > > >> > >> >> > > discussion > > > > > > > >> > >> >> > > > > > > thread > > > > > > > >> > >> >> > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > for > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > >> > >> >> > > > > > > where > > > > > > > >> > >> >> > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > memory > > > > > > > >> > >> >> > > configurations. > > > > > > > >> > >> >> > > > > The > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > early > > > design > > > > > > > "Memory > > > > > > > >> > >> >> Management > > > > > > > >> > >> >> > > and > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > with > > > updates > > > > > > from > > > > > > > >> > >> follow-up > > > > > > > >> > >> >> > > > > discussions > > > > > > > >> > >> >> > > > > > > > both > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > This > > FLIP > > > > > > > addresses > > > > > > > >> > >> several > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > memory > > > > > > > >> > >> >> > > configuration. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Different > > > > > > > >> > >> configuration > > > > > > > >> > >> >> > for > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > Complex > > > > > and > > > > > > > >> > >> difficult > > > > > > > >> > >> >> > > > > > configuration > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Complicated, > > > > > > > >> > >> uncertain > > > > > > > >> > >> >> and > > > > > > > >> > >> >> > > > hard > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key > > > changes > > > > to > > > > > > > solve > > > > > > > >> > the > > > > > > > >> > >> >> > problems > > > > > > > >> > >> >> > > > can > > > > > > > >> > >> >> > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > Extend > > > > > > memory > > > > > > > >> > >> manager > > > > > > > >> > >> >> to > > > > > > > >> > >> >> > > also > > > > > > > >> > >> >> > > > > > > account > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > Modify > > > > > how > > > > > > > >> > >> TaskExecutor > > > > > > > >> > >> >> > > memory > > > > > > > >> > >> >> > > > > is > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > individual > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > >> reservations > > > > > > > >> > >> and > > > > > > > >> > >> >> > pools. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Simplify > > > > > > > memory > > > > > > > >> > >> >> > > configuration > > > > > > > >> > >> >> > > > > > > options > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Please > > > find > > > > > more > > > > > > > >> > details > > > > > > > >> > >> in > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > FLIP > > > > > > > >> > >> >> > > > > > wiki > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > note > > > > > > that > > > > > > > >> the > > > > > > > >> > >> early > > > > > > > >> > >> >> > > design > > > > > > > >> > >> >> > > > > doc > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > >> > >> >> > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > to > > > > > > > have > > > > > > > >> the > > > > > > > >> > >> >> > > discussion > > > > > > > >> > >> >> > > > in > > > > > > > >> > >> >> > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > forward > > > > > > to > > > > > > > >> your > > > > > > > >> > >> >> > > feedbacks. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank > > you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at > 2:16 > > > PM > > > > > > > Xintong > > > > > > > >> > Song > > > > > > > >> > >> < > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing your > > > > opinion > > > > > > > Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor of > > > > alternative > > > > > > 2. > > > > > > > I > > > > > > > >> was > > > > > > > >> > >> >> > wondering > > > > > > > >> > >> >> > > > > > whether > > > > > > > >> > >> >> > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > >> > >> >> > > > > > > > > > > > > > > using Unsafe.allocate() > > for > > > > > > off-heap > > > > > > > >> > >> managed > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > > and > > > > > > > >> > >> >> > > > > > > > network > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But after > > > > giving > > > > > > it a > > > > > > > >> > second > > > > > > > >> > >> >> > > thought, > > > > > > > >> > >> >> > > > I > > > > > > > >> > >> >> > > > > > > think > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 using > direct > > > > > memory > > > > > > > for > > > > > > > >> > >> off-heap > > > > > > > >> > >> >> > > > managed > > > > > > > >> > >> >> > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your concern, > I > > > > think > > > > > > what > > > > > > > >> > >> proposed > > > > > > > >> > >> >> in > > > > > > > >> > >> >> > > this > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > >> > >> >> > > > > > > it > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed memory > > and > > > > > > network > > > > > > > >> > memory > > > > > > > >> > >> >> > > allocated > > > > > > > >> > >> >> > > > > > > through > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), which > > > means > > > > > > they > > > > > > > >> are > > > > > > > >> > >> >> > practically > > > > > > > >> > >> >> > > > > > native > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > direct > > > > > memory. > > > > > > > The > > > > > > > >> > only > > > > > > > >> > >> >> parts > > > > > > > >> > >> >> > of > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory are > task > > > > > > off-heap > > > > > > > >> > memory > > > > > > > >> > >> and > > > > > > > >> > >> >> > JVM > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly alternative 2 > > > suggests > > > > > to > > > > > > > set > > > > > > > >> the > > > > > > > >> > >> JVM > > > > > > > >> > >> >> max > > > > > > > >> > >> >> > > > > direct > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at > > 1:48 > > > > PM > > > > > > Till > > > > > > > >> > >> Rohrmann > > > > > > > >> > >> >> < > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > clarification > > > > > > > >> Xintong. I > > > > > > > >> > >> >> > > understand > > > > > > > >> > >> >> > > > > the > > > > > > > >> > >> >> > > > > > > two > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in favour > of > > > > > option 2 > > > > > > > >> > because > > > > > > > >> > >> it > > > > > > > >> > >> >> > makes > > > > > > > >> > >> >> > > > > > things > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the direct > > > > > memory, I > > > > > > > >> fear > > > > > > > >> > >> that > > > > > > > >> > >> >> we > > > > > > > >> > >> >> > > might > > > > > > > >> > >> >> > > > > end > > > > > > > >> > >> >> > > > > > > up > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we are > > > > currently > > > > > > in: > > > > > > > >> The > > > > > > > >> > >> user > > > > > > > >> > >> >> > might > > > > > > > >> > >> >> > > > see > > > > > > > >> > >> >> > > > > > that > > > > > > > >> > >> >> > > > > > > > her > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS and > > does > > > > not > > > > > > know > > > > > > > >> why > > > > > > > >> > >> this > > > > > > > >> > >> >> is > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > case. > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease the > > > > process > > > > > > > memory > > > > > > > >> > size > > > > > > > >> > >> >> > > (similar > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order to > > > > accommodate > > > > > > for > > > > > > > >> the > > > > > > > >> > >> extra > > > > > > > >> > >> >> > > direct > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > memory > > > > > budgets > > > > > > > >> which > > > > > > > >> > >> are > > > > > > > >> > >> >> not > > > > > > > >> > >> >> > > > fully > > > > > > > >> > >> >> > > > > > used > > > > > > > >> > >> >> > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the overall > > memory > > > > > > > >> consumption. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > at > > > > 11:01 > > > > > AM > > > > > > > >> > Xintong > > > > > > > >> > >> >> Song < > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain this > > > with a > > > > > > > >> concrete > > > > > > > >> > >> >> example > > > > > > > >> > >> >> > > Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we have > the > > > > > > following > > > > > > > >> > >> scenario. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > Memory: > > > 1GB > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct Memory > > (Task > > > > > > Off-Heap > > > > > > > >> > >> Memory + > > > > > > > >> > >> >> JVM > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory (JVM > Heap > > > > > Memory, > > > > > > > JVM > > > > > > > >> > >> >> Metaspace, > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network Memory): > 800MB > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 2, > we > > > set > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 3, > we > > > set > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > a > > > > > > > >> > >> >> > > > > > > > very > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual direct > > > > memory > > > > > > > usage > > > > > > > >> of > > > > > > > >> > >> Task > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed 200MB, > > > then > > > > > > > >> > alternative 2 > > > > > > > >> > >> >> and > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > Setting > > > > larger > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > > > > will > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the other > > > memory > > > > > > pools. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual direct > > > > memory > > > > > > > usage > > > > > > > >> of > > > > > > > >> > >> Task > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead potentially > > > > exceed > > > > > > > 200MB, > > > > > > > >> > then > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - Alternative 2 > > > suffers > > > > > > from > > > > > > > >> > >> frequent > > > > > > > >> > >> >> OOM. > > > > > > > >> > >> >> > > To > > > > > > > >> > >> >> > > > > > avoid > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do is to > > > > modify > > > > > > the > > > > > > > >> > >> >> configuration > > > > > > > >> > >> >> > > and > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task Off-Heap > > > Memory + > > > > > JVM > > > > > > > >> > >> Overhead). > > > > > > > >> > >> >> > Let's > > > > > > > >> > >> >> > > > say > > > > > > > >> > >> >> > > > > > > that > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct Memory to > > > 250MB, > > > > > > this > > > > > > > >> will > > > > > > > >> > >> >> reduce > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > > total > > > > > > > >> > >> >> > > > > > > > size > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to 750MB, > > given > > > > the > > > > > > > total > > > > > > > >> > >> process > > > > > > > >> > >> >> > > memory > > > > > > > >> > >> >> > > > > > > remains > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For alternative > > 3, > > > > > there > > > > > > is > > > > > > > >> no > > > > > > > >> > >> >> chance of > > > > > > > >> > >> >> > > > > direct > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of exceeding the > > > total > > > > > > > process > > > > > > > >> > >> memory > > > > > > > >> > >> >> > limit, > > > > > > > >> > >> >> > > > but > > > > > > > >> > >> >> > > > > > > given > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up all > the > > > > > reserved > > > > > > > >> native > > > > > > > >> > >> >> memory > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > Metaspace), > > > > > if > > > > > > > the > > > > > > > >> > >> actual > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very close to > > > > 200MB, > > > > > > user > > > > > > > >> > >> probably > > > > > > > >> > >> >> do > > > > > > > >> > >> >> > > not > > > > > > > >> > >> >> > > > > need > > > > > > > >> > >> >> > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > configurations. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I think > > from > > > > the > > > > > > > user's > > > > > > > >> > >> >> > > perspective, a > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for alternative 2 > may > > > lead > > > > > to > > > > > > > >> lower > > > > > > > >> > >> >> resource > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > > 10:28 > > > > > > AM > > > > > > > >> Till > > > > > > > >> > >> >> > Rohrmann > > > > > > > >> > >> >> > > < > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you have > to > > > help > > > > > me > > > > > > > >> > >> understand > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > > > difference > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > memory > > > > under > > > > > > > >> > utilization > > > > > > > >> > >> >> > > Xintong. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: > set > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > Task > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. Then > there > > > is > > > > > the > > > > > > > risk > > > > > > > >> > that > > > > > > > >> > >> >> this > > > > > > > >> > >> >> > > size > > > > > > > >> > >> >> > > > > is > > > > > > > >> > >> >> > > > > > > too > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of garbage > > > > collection > > > > > > and > > > > > > > >> > >> >> potentially > > > > > > > >> > >> >> > an > > > > > > > >> > >> >> > > > OOM. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 3: > set > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > > > > something > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 2. > This > > > > would > > > > > of > > > > > > > >> course > > > > > > > >> > >> >> reduce > > > > > > > >> > >> >> > > the > > > > > > > >> > >> >> > > > > > sizes > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > alternative > > > 2 > > > > > now > > > > > > > >> result > > > > > > > >> > >> in an > > > > > > > >> > >> >> > > under > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > alternative > > > > 3? > > > > > > If > > > > > > > >> > >> >> alternative 3 > > > > > > > >> > >> >> > > > > > strictly > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct memory size > > and > > > > we > > > > > > use > > > > > > > >> only > > > > > > > >> > >> >> little, > > > > > > > >> > >> >> > > > then I > > > > > > > >> > >> >> > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 3 > > results > > > in > > > > > > > memory > > > > > > > >> > under > > > > > > > >> > >> >> > > > > utilization. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, > 2019 > > > at > > > > > 4:19 > > > > > > > PM > > > > > > > >> > Yang > > > > > > > >> > >> >> Wang < > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi xintong,till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native and > > Direct > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point is > > setting > > > a > > > > > very > > > > > > > >> large > > > > > > > >> > >> max > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > size > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > differentiate > > direct > > > > and > > > > > > > >> native > > > > > > > >> > >> >> memory. > > > > > > > >> > >> >> > If > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct memory > and > > > > > > framework > > > > > > > >> > direct > > > > > > > >> > >> >> > > > memory,could > > > > > > > >> > >> >> > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in favor of > > > > setting > > > > > > > >> direct > > > > > > > >> > >> memory > > > > > > > >> > >> >> > with > > > > > > > >> > >> >> > > > > fixed > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > Calculation > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree with > > > xintong. > > > > > For > > > > > > > Yarn > > > > > > > >> > and > > > > > > > >> > >> >> k8s,we > > > > > > > >> > >> >> > > > need > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > check > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configurations > in > > > > client > > > > > > to > > > > > > > >> avoid > > > > > > > >> > >> >> > > submitting > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > master. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < > > > > > > > >> > >> [hidden email] > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks for > > > replying, > > > > > > Till. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > MemorySegment, I > > > > > > > think > > > > > > > >> > you > > > > > > > >> > >> are > > > > > > > >> > >> >> > > right > > > > > > > >> > >> >> > > > > that > > > > > > > >> > >> >> > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in the > > scope > > > > of > > > > > > this > > > > > > > >> > FLIP. > > > > > > > >> > >> >> This > > > > > > > >> > >> >> > > FLIP > > > > > > > >> > >> >> > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configure > memory > > > > pools > > > > > > for > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > >> > >> >> > > > > with > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > consumers > > > use > > > > > it. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About direct > > > > memory, I > > > > > > > think > > > > > > > >> > >> >> > alternative > > > > > > > >> > >> >> > > 3 > > > > > > > >> > >> >> > > > > may > > > > > > > >> > >> >> > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > reservation > > issue > > > > that > > > > > > > >> > >> alternative 2 > > > > > > > >> > >> >> > > does, > > > > > > > >> > >> >> > > > > but > > > > > > > >> > >> >> > > > > > at > > > > > > > >> > >> >> > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using memory > at > > > the > > > > > > > >> container > > > > > > > >> > >> level, > > > > > > > >> > >> >> > > which > > > > > > > >> > >> >> > > > is > > > > > > > >> > >> >> > > > > > not > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both "Task > > > Off-Heap > > > > > > > Memory" > > > > > > > >> and > > > > > > > >> > >> "JVM > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > >> > >> >> > > > > > > are > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative 2, > > > users > > > > > > might > > > > > > > >> > >> configure > > > > > > > >> > >> >> > them > > > > > > > >> > >> >> > > > > > higher > > > > > > > >> > >> >> > > > > > > > than > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to avoid > > > > getting > > > > > a > > > > > > > >> direct > > > > > > > >> > >> OOM. > > > > > > > >> > >> >> For > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct OOM, so > > > they > > > > > may > > > > > > > not > > > > > > > >> > >> config > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > two > > > > > > > >> > >> >> > > > > > > options > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > consequences > > > are > > > > > > risks > > > > > > > >> of > > > > > > > >> > >> >> overall > > > > > > > >> > >> >> > > > > container > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug > 13, > > > 2019 > > > > > at > > > > > > > >> 9:39 AM > > > > > > > >> > >> Till > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > >> > >> >> > > > > > > > > > > > > > > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for > > > > proposing > > > > > > > this > > > > > > > >> > FLIP > > > > > > > >> > >> >> > Xintong. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in all I > > > think > > > > > it > > > > > > > >> already > > > > > > > >> > >> >> looks > > > > > > > >> > >> >> > > quite > > > > > > > >> > >> >> > > > > > good. > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > question > about > > > > > > > allocating > > > > > > > >> > >> memory > > > > > > > >> > >> >> > > > segments, > > > > > > > >> > >> >> > > > > I > > > > > > > >> > >> >> > > > > > > was > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > strictly > > > necessary > > > > > to > > > > > > do > > > > > > > >> in > > > > > > > >> > the > > > > > > > >> > >> >> > context > > > > > > > >> > >> >> > > > of > > > > > > > >> > >> >> > > > > > this > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be done as a > > > > follow > > > > > > up? > > > > > > > >> > Without > > > > > > > >> > >> >> > knowing > > > > > > > >> > >> >> > > > all > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that we > would > > > > widen > > > > > > the > > > > > > > >> scope > > > > > > > >> > >> of > > > > > > > >> > >> >> this > > > > > > > >> > >> >> > > > FLIP > > > > > > > >> > >> >> > > > > > too > > > > > > > >> > >> >> > > > > > > > much > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to touch all > > the > > > > > > > existing > > > > > > > >> > call > > > > > > > >> > >> >> sites > > > > > > > >> > >> >> > of > > > > > > > >> > >> >> > > > the > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > segments > > > > > (this > > > > > > > >> should > > > > > > > >> > >> >> mainly > > > > > > > >> > >> >> > be > > > > > > > >> > >> >> > > > > batch > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the memory > > > > > reservation > > > > > > > >> call > > > > > > > >> > to > > > > > > > >> > >> the > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this and I > > would > > > > > hope > > > > > > > that > > > > > > > >> > >> this is > > > > > > > >> > >> >> > the > > > > > > > >> > >> >> > > > only > > > > > > > >> > >> >> > > > > > > point > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > streaming > job > > > > would > > > > > > have > > > > > > > >> with > > > > > > > >> > >> the > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Concerning > the > > > > > second > > > > > > > open > > > > > > > >> > >> >> question > > > > > > > >> > >> >> > > about > > > > > > > >> > >> >> > > > > > > setting > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > memory > > > > > limit, I > > > > > > > >> would > > > > > > > >> > >> also > > > > > > > >> > >> >> be > > > > > > > >> > >> >> > > > > > interested > > > > > > > >> > >> >> > > > > > > > why > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > leaving it > > open > > > > > would > > > > > > be > > > > > > > >> > best. > > > > > > > >> > >> My > > > > > > > >> > >> >> > > concern > > > > > > > >> > >> >> > > > > > about > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in a > > similar > > > > > > > situation > > > > > > > >> as > > > > > > > >> > we > > > > > > > >> > >> >> are > > > > > > > >> > >> >> > now > > > > > > > >> > >> >> > > > > with > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > different > > > > memory > > > > > > > pools > > > > > > > >> > are > > > > > > > >> > >> not > > > > > > > >> > >> >> > > > clearly > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a different > > > pool, > > > > > then > > > > > > > it > > > > > > > >> is > > > > > > > >> > >> quite > > > > > > > >> > >> >> > hard > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > process to > get > > > > > killed > > > > > > > for > > > > > > > >> > using > > > > > > > >> > >> >> too > > > > > > > >> > >> >> > > much > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead to a > > > similar > > > > > > > >> situation > > > > > > > >> > >> what > > > > > > > >> > >> >> we > > > > > > > >> > >> >> > > have > > > > > > > >> > >> >> > > > > with > > > > > > > >> > >> >> > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > setting a > sane > > > > > default > > > > > > > >> value > > > > > > > >> > >> for > > > > > > > >> > >> >> max > > > > > > > >> > >> >> > > > direct > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option to > > > increase > > > > > it > > > > > > if > > > > > > > >> he > > > > > > > >> > >> runs > > > > > > > >> > >> >> into > > > > > > > >> > >> >> > > an > > > > > > > >> > >> >> > > > > OOM. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, > how > > > > would > > > > > > > >> > >> alternative 2 > > > > > > > >> > >> >> > lead > > > > > > > >> > >> >> > > to > > > > > > > >> > >> >> > > > > > lower > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > alternative > 3 > > > > where > > > > > we > > > > > > > set > > > > > > > >> > the > > > > > > > >> > >> >> direct > > > > > > > >> > >> >> > > > > memory > > > > > > > >> > >> >> > > > > > > to a > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, Aug > 9, > > > > 2019 > > > > > at > > > > > > > >> 9:12 > > > > > > > >> > AM > > > > > > > >> > >> >> > Xintong > > > > > > > >> > >> >> > > > > Song < > > > > > > > >> > >> >> > > > > > > > > > > > > > > > [hidden email] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks for > > the > > > > > > > feedback, > > > > > > > >> > >> Yang. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Regarding > > your > > > > > > > comments: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Native > and > > > > Direct > > > > > > > >> Memory* > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > > setting > > > > a > > > > > > very > > > > > > > >> > large > > > > > > > >> > >> max > > > > > > > >> > >> >> > > direct > > > > > > > >> > >> >> > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > sides. > > > > E.g., > > > > > we > > > > > > > do > > > > > > > >> not > > > > > > > >> > >> >> worry > > > > > > > >> > >> >> > > about > > > > > > > >> > >> >> > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > allocate > > > > > managed > > > > > > / > > > > > > > >> > network > > > > > > > >> > >> >> > memory > > > > > > > >> > >> >> > > > with > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > However, > > there > > > > are > > > > > > > also > > > > > > > >> > some > > > > > > > >> > >> >> down > > > > > > > >> > >> >> > > sides > > > > > > > >> > >> >> > > > > of > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - One > > > thing I > > > > > can > > > > > > > >> think > > > > > > > >> > >> of is > > > > > > > >> > >> >> > that > > > > > > > >> > >> >> > > > if > > > > > > > >> > >> >> > > > > a > > > > > > > >> > >> >> > > > > > > task > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > killed > > due > > > to > > > > > > > >> overusing > > > > > > > >> > >> >> memory, > > > > > > > >> > >> >> > it > > > > > > > >> > >> >> > > > > could > > > > > > > >> > >> >> > > > > > > be > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > of the > > > memory > > > > > is > > > > > > > >> > overused. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > Another > > > > down > > > > > > side > > > > > > > >> is > > > > > > > >> > >> that > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > JVM > > > > > > > >> > >> >> > > > > > never > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > direct > > > memory > > > > > > > limit, > > > > > > > >> > >> because > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > limit > > > > > > > >> > >> >> > > > > > is > > > > > > > >> > >> >> > > > > > > > too > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > means > we > > > kind > > > > > of > > > > > > > >> relay > > > > > > > >> > on > > > > > > > >> > >> >> heap > > > > > > > >> > >> >> > > > memory > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory. > > > That > > > > > > could > > > > > > > >> be a > > > > > > > >> > >> >> problem > > > > > > > >> > >> >> > in > > > > > > > >> > >> >> > > > > cases > > > > > > > >> > >> >> > > > > > > > where > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > usage > but > > > not > > > > > > > enough > > > > > > > >> > heap > > > > > > > >> > >> >> > activity > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe you > > can > > > > > share > > > > > > > your > > > > > > > >> > >> reasons > > > > > > > >> > >> >> > for > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if there > are > > > > > > anything > > > > > > > >> else > > > > > > > >> > I > > > > > > > >> > >> >> > > > overlooked. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > Calculation* > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If there > is > > > any > > > > > > > conflict > > > > > > > >> > >> between > > > > > > > >> > >> >> > > > multiple > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > specified, > > > > > > > I > > > > > > > >> > >> think we > > > > > > > >> > >> >> > > should > > > > > > > >> > >> >> > > > > > throw > > > > > > > >> > >> >> > > > > > > > an > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > doing > > > > > > checking > > > > > > > >> on > > > > > > > >> > the > > > > > > > >> > >> >> > client > > > > > > > >> > >> >> > > > side > > > > > > > >> > >> >> > > > > > is > > > > > > > >> > >> >> > > > > > > a > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we can > > > > > discover > > > > > > > the > > > > > > > >> > >> problem > > > > > > > >> > >> >> > > before > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is always > a > > > good > > > > > > > thing. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But we can > > not > > > > > only > > > > > > > >> rely on > > > > > > > >> > >> the > > > > > > > >> > >> >> > > client > > > > > > > >> > >> >> > > > > side > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > standalone > > > > cluster > > > > > > > >> > >> TaskManagers > > > > > > > >> > >> >> on > > > > > > > >> > >> >> > > > > > different > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > different > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > and > > > > > > the > > > > > > > >> > client > > > > > > > >> > >> >> does > > > > > > > >> > >> >> > > see > > > > > > > >> > >> >> > > > > > that. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What do > you > > > > think? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong > Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, > Aug > > 8, > > > > > 2019 > > > > > > at > > > > > > > >> 5:09 > > > > > > > >> > >> PM > > > > > > > >> > >> >> Yang > > > > > > > >> > >> >> > > > Wang > > > > > > > >> > >> >> > > > > < > > > > > > > >> > >> >> > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > xintong, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > for > > > > your > > > > > > > >> detailed > > > > > > > >> > >> >> > proposal. > > > > > > > >> > >> >> > > > > After > > > > > > > >> > >> >> > > > > > > all > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > it > > > > > > will > > > > > > > be > > > > > > > >> > more > > > > > > > >> > >> >> > > powerful > > > > > > > >> > >> >> > > > to > > > > > > > >> > >> >> > > > > > > > control > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > just > have > > > few > > > > > > > >> questions > > > > > > > >> > >> about > > > > > > > >> > >> >> it. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > Native > > > > and > > > > > > > Direct > > > > > > > >> > >> Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We do > not > > > > > > > >> differentiate > > > > > > > >> > >> user > > > > > > > >> > >> >> > direct > > > > > > > >> > >> >> > > > > > memory > > > > > > > >> > >> >> > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > included > > in > > > > task > > > > > > > >> off-heap > > > > > > > >> > >> >> memory. > > > > > > > >> > >> >> > > > > Right? > > > > > > > >> > >> >> > > > > > > So i > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > >> > >> >> > > > properly. I > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > Memory > > > > > > > >> Calculation > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If the > sum > > > of > > > > > and > > > > > > > >> > >> fine-grained > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > larger > > > than > > > > > > total > > > > > > > >> > >> process > > > > > > > >> > >> >> > > memory, > > > > > > > >> > >> >> > > > > how > > > > > > > >> > >> >> > > > > > do > > > > > > > >> > >> >> > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we need > to > > > > check > > > > > > the > > > > > > > >> > memory > > > > > > > >> > >> >> > > > > configuration > > > > > > > >> > >> >> > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong > > > Song < > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > everyone, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > would > > > > like > > > > > to > > > > > > > >> start > > > > > > > >> > a > > > > > > > >> > >> >> > > discussion > > > > > > > >> > >> >> > > > > > > thread > > > > > > > >> > >> >> > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > for > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > >> > >> >> > > > > > > where > > > > > > > >> > >> >> > > > > > > > we > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > memory > > > > > > > >> > >> >> > > configurations. > > > > > > > >> > >> >> > > > > The > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > early > > > design > > > > > > > "Memory > > > > > > > >> > >> >> Management > > > > > > > >> > >> >> > > and > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > with > > > updates > > > > > > from > > > > > > > >> > >> follow-up > > > > > > > >> > >> >> > > > > discussions > > > > > > > >> > >> >> > > > > > > > both > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > This > > FLIP > > > > > > > addresses > > > > > > > >> > >> several > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > memory > > > > > > > >> > >> >> > > configuration. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Different > > > > > > > >> > >> configuration > > > > > > > >> > >> >> > for > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > Complex > > > > > and > > > > > > > >> > >> difficult > > > > > > > >> > >> >> > > > > > configuration > > > > > > > >> > >> >> > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Complicated, > > > > > > > >> > >> uncertain > > > > > > > >> > >> >> and > > > > > > > >> > >> >> > > > hard > > > > > > > >> > >> >> > > > > to > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key > > > changes > > > > to > > > > > > > solve > > > > > > > >> > the > > > > > > > >> > >> >> > problems > > > > > > > >> > >> >> > > > can > > > > > > > >> > >> >> > > > > > be > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > Extend > > > > > > memory > > > > > > > >> > >> manager > > > > > > > >> > >> >> to > > > > > > > >> > >> >> > > also > > > > > > > >> > >> >> > > > > > > account > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > Modify > > > > > how > > > > > > > >> > >> TaskExecutor > > > > > > > >> > >> >> > > memory > > > > > > > >> > >> >> > > > > is > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > individual > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > >> reservations > > > > > > > >> > >> and > > > > > > > >> > >> >> > pools. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Simplify > > > > > > > memory > > > > > > > >> > >> >> > > configuration > > > > > > > >> > >> >> > > > > > > options > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Please > > > find > > > > > more > > > > > > > >> > details > > > > > > > >> > >> in > > > > > > > >> > >> >> the > > > > > > > >> > >> >> > > > FLIP > > > > > > > >> > >> >> > > > > > wiki > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > note > > > > > > that > > > > > > > >> the > > > > > > > >> > >> early > > > > > > > >> > >> >> > > design > > > > > > > >> > >> >> > > > > doc > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > >> > >> >> > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > to > > > > > > > have > > > > > > > >> the > > > > > > > >> > >> >> > > discussion > > > > > > > >> > >> >> > > > in > > > > > > > >> > >> >> > > > > > > this > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > forward > > > > > > to > > > > > > > >> your > > > > > > > >> > >> >> > > feedbacks. > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank > > you~ > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > Song > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
|
I think conceptually it makes sense to have a MemoryManager per slot. As
Andrey said, it should not add any noticeable overhead. Moreover, I think this model does not contradict any of the mentioned future improvements. For example, a per slot MemoryManager could still be implemented such that the TaskExecutor's MemoryManager keeps a reference to it and can request to give up/assign more memory segments in case of revocable memory. The benefit I see is a better abstraction and hence a more targeted component which will be easier to maintain and evolve. Cheers, Till On Mon, Sep 16, 2019 at 11:23 AM Xintong Song <[hidden email]> wrote: > I would say both ways (one unified memory manager per task executor, or one > isolated memory manager per slot) should work well at the moment. I've not > yet think this through very detailedly, but just from what I can think of > at the moment, I'm in favor of having one memory manager per task executor, > for the following reasons: > > - We need the memory manager to understand quota of each operator anyway > (with the provided fractions), for limiting the memory usage of > un-revocable operators. It shouldn't be much more complicated for the > memory manager to isolate memory across slots. Actually I'm wondering > wether it requires more efforts at all compare to having one memory manager > per slot, where we also need to handle the creation and destroying of > memory managers. > > - It is true that we have no plan for segments caching at the moment. But I > would say it is a good opportunity for future improvement. E.g., we may > have some "lazy de-allocation" strategy that only de-allocate segments if > the remaining available memory quota cannot satisfy new coming reservation > requests. I understand that we should not complicate the current design for > any "potential future features". My concern is that we make efforts > changing from one memory manager per task executor to one per slot, with > the benefit (easier to isolate memory across slots) that is uncertain to > me, and later discover that we may want to change it back for further > optimizations. > > Anyway, I do not have strong objection against having one memory manager > per slot. Just raising my concerns. > > Thank you~ > > Xintong Song > > > > On Mon, Sep 16, 2019 at 4:16 PM Andrey Zagrebin <[hidden email]> > wrote: > > > @Xintong > > > > Just to understand, what kind of overhead do you mean by creating the > > memory manager for every dynamic slot? > > We agreed not to preallocate segments anymore and there was no plan to do > > any segment caching at the moment, AFAIK. > > This means the segments are just garbage collected upon release and not > > reused. > > If we decide to go with caching or revoking later and share pools among > > slots, it would already mean mixed segment/reservation types of memory > > allocation in general (like batch and streaming). > > Then I think memory manager would need some input from tasks because it > is > > hard for Flink to decide how much to cache and which pool to allocate > from > > eventually. > > As I understand, tasks do not really need this now. That is why I think > > having memory manager per slot will give more security by isolating slots > > with relatively small effort. > > If there is no preallocation/caching then creating the memory manager is > > almost no-op, later we can reconsider it if needed. What do you think? > > > > Thanks, > > Andrey > > > > On Sat, Sep 14, 2019 at 8:11 AM Xintong Song <[hidden email]> > > wrote: > > > > > @Andrey, > > > > > > If we speak in the condition of dynamic slot allocation, then DataSet > > tasks > > > should always request slots with unknown resource profiles. Then the > > > allocated slots should have 1/n on-heap managed memory and 1/n off-heap > > > managed memory of the task executor, where n is the configured > > > 'numberOfSlot'. Given that DataSet operators only allocate segments (no > > > reservation), and does not care whether the segments are on-heap or > > > off-heap, I think it makes sense to wrapping both pools for DataSet > > > operators, with respect to the on-heap/off-heap quota of the allocated > > > slot. > > > > > > I'm not sure whether we want to create one memory manager per slot. I > > think > > > we can have limit the quota of each slot with one memory manager per > task > > > executor. Given the dynamic slot allocation that dynamically create and > > > destroy slots, I think having one memory manager per task executor may > > save > > > the overhead for frequently creating and destroying memory managers, > and > > > provide the chance for reusing memory segments across slots. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Fri, Sep 13, 2019 at 11:47 PM Andrey Zagrebin <[hidden email] > > > > > wrote: > > > > > > > Hi Xintong, > > > > > > > > True, there would be no regression if only one type of memory is > > > > configured. This can be a problem only for the old jobs running in a > > > newly > > > > configured cluster. > > > > > > > > About the pool type precedence, in general, it should not matter for > > the > > > > users which type the segments have. > > > > The first implementation can be just to pull from any pool, e.g. > empty > > > one > > > > pool firstly and then another or some other random pulling. > > > > This might be a problem if we mix segment allocations and reservation > > of > > > > memory chunks from the same memory manager. > > > > The reservation will be usually for a certain type of memory then the > > > task > > > > will probably have to also decide from which pool to allocate the > > > segments. > > > > I would suggest we create a memory manager per slot and give it > memory > > > > limit of the slot then we do not have this kind of mixed operation > > > > because Dataset/Batch jobs need only segment memory allocations and > > > > streaming jobs need only memory chunks for state backends as I > > understand > > > > the current plan. > > > > I would suggest we will look at it if we have the mixed operations at > > > some > > > > point and it becomes a problem. > > > > > > > > Thanks, > > > > Andrey > > > > > > > > On Fri, Sep 13, 2019 at 5:24 PM Andrey Zagrebin < > [hidden email]> > > > > wrote: > > > > > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > From: Xintong Song <[hidden email]> > > > > > Date: Thu, Sep 12, 2019 at 4:21 AM > > > > > Subject: Re: [DISCUSS] FLIP-49: Unified Memory Configuration for > > > > > TaskExecutors > > > > > To: dev <[hidden email]> > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > Thanks for bringing this up. > > > > > > > > > > If I understand correctly, this issue only occurs where the cluster > > is > > > > > configured with both on-heap and off-heap memory. There should be > no > > > > > regression for clusters configured in the old way (either all > on-heap > > > or > > > > > all off-heap). > > > > > > > > > > I also agree that it would be good if the DataSet API jobs can use > > both > > > > > memory types. The only question I can see is that, from which pool > > > (heap > > > > / > > > > > off-heap) should we allocate memory for DataSet API operators? Do > we > > > > > always prioritize one pool over the other? Or do we always > prioritize > > > the > > > > > pool with more available memory left? > > > > > > > > > > Thank you~ > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 8:15 PM Andrey Zagrebin < > > [hidden email]> > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > While looking more into the implementation details of Step 4, we > > > > released > > > > > > during some offline discussions with @Till > > > > > > that there can be a performance degradation for the batch DataSet > > API > > > > if > > > > > we > > > > > > simply continue to pull memory from the pool > > > > > > according the legacy option taskmanager.memory.off-heap. > > > > > > > > > > > > The reason is that if the cluster is newly configured to > statically > > > > split > > > > > > heap/off-heap (not like previously either heap or 0ff-heap) > > > > > > then the batch DataSet API jobs will be able to use only one type > > of > > > > > > memory. Although it does not really matter where the memory > > segments > > > > come > > > > > > from > > > > > > and potentially batch jobs can use both. Also, currently the > > Dataset > > > > API > > > > > > does not result in absolute resource requirements and its batch > > jobs > > > > will > > > > > > always get a default share of TM resources. > > > > > > > > > > > > The suggestion is that we let the batch tasks of Dataset API pull > > > from > > > > > both > > > > > > pools according to their fair slot share of each memory type. > > > > > > For that we can have a special wrapping view of both pools which > > will > > > > > pull > > > > > > segments (can be randomly) according to the slot limits. > > > > > > The view can wrap TM level memory pools and be given to the Task. > > > > > > > > > > > > Best, > > > > > > Andrey > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:35 PM Xintong Song < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > Thanks for your comments, Andrey. > > > > > > > > > > > > > > - Regarding Task Off-Heap Memory, I think you're right that the > > > user > > > > > need > > > > > > > to make sure that direct memory and native memory together used > > by > > > > the > > > > > > user > > > > > > > code (external libs) do not exceed the configured value. As far > > as > > > I > > > > > can > > > > > > > think of, there is nothing we can do about it. > > > > > > > > > > > > > > I addressed the rest of your comment in the wiki page [1]. > Please > > > > take > > > > > a > > > > > > > look. > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 6:13 PM Andrey Zagrebin < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > EDIT: sorry for confusion I meant > > > > > > > > taskmanager.memory.off-heap > > > > > > > > instead of > > > > > > > > setting taskmanager.memory.preallocate > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 11:29 AM Andrey Zagrebin < > > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > @Xitong thanks a lot for driving the discussion. > > > > > > > > > > > > > > > > > > I also reviewed the FLIP and it looks quite good to me. > > > > > > > > > Here are some comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I wanted to discuss is the > > > backwards-compatibility > > > > > > with > > > > > > > > > the previous user setups. We could list which options we > > > plan > > > > to > > > > > > > > deprecate. > > > > > > > > > From the first glance it looks possible to provide the > > > > > > same/similar > > > > > > > > > behaviour for the setups relying on the deprecated > > options. > > > > E.g. > > > > > > > > > setting taskmanager.memory.preallocate to true could > > > override > > > > > the > > > > > > > > > new taskmanager.memory.managed.offheap-fraction to 1 > etc. > > At > > > > the > > > > > > > > moment the > > > > > > > > > FLIP just states that in some cases it may require > > > > > re-configuring > > > > > > of > > > > > > > > > cluster if migrated from prior versions. My suggestion > is > > > that > > > > > we > > > > > > > try > > > > > > > > to > > > > > > > > > keep it backwards-compatible unless there is a good > reason > > > > like > > > > > > some > > > > > > > > major > > > > > > > > > complication for the implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > Also couple of smaller things: > > > > > > > > > > > > > > > > > > - I suggest we remove TaskExecutorSpecifics from the > FLIP > > > and > > > > > > leave > > > > > > > > > some general wording atm, like 'data structure to store' > > or > > > > > > 'utility > > > > > > > > > classes'. When the classes are implemented, we put the > > > > concrete > > > > > > > class > > > > > > > > > names. This way we can avoid confusion and stale > > documents. > > > > > > > > > > > > > > > > > > > > > > > > > > > - As I understand, if user task uses native memory (not > > > direct > > > > > > > memory, > > > > > > > > > but e.g. unsafe.allocate or from external lib), there > will > > > be > > > > no > > > > > > > > > explicit guard against exceeding 'task off heap memory'. > > > Then > > > > > user > > > > > > > > should > > > > > > > > > still explicitly make sure that her/his direct buffer > > > > allocation > > > > > > > plus > > > > > > > > any > > > > > > > > > other memory usages does not exceed value announced as > > 'task > > > > off > > > > > > > > heap'. I > > > > > > > > > guess there is no so much that can be done about it > except > > > > > > > mentioning > > > > > > > > in > > > > > > > > > docs, similar to controlling the heap state backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 10:07 AM Yang Wang < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> I also agree that all the configuration should be > calculated > > > out > > > > > of > > > > > > > > >> TaskManager. > > > > > > > > >> > > > > > > > > >> So a full configuration should be generated before > > TaskManager > > > > > > > started. > > > > > > > > >> > > > > > > > > >> Override the calculated configurations through -D now > seems > > > > > better. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> Best, > > > > > > > > >> > > > > > > > > >> Yang > > > > > > > > >> > > > > > > > > >> Xintong Song <[hidden email]> 于2019年9月2日周一 > > 上午11:39写道： > > > > > > > > >> > > > > > > > > >> > I just updated the FLIP wiki page [1], with the > following > > > > > changes: > > > > > > > > >> > > > > > > > > > >> > - Network memory uses JVM direct memory, and is > > accounted > > > > > when > > > > > > > > >> setting > > > > > > > > >> > JVM max direct memory size parameter. > > > > > > > > >> > - Use dynamic configurations (`-Dkey=value`) to pass > > > > > calculated > > > > > > > > >> memory > > > > > > > > >> > configs into TaskExecutors, instead of ENV variables. > > > > > > > > >> > - Remove 'supporting memory reservation' from the > scope > > > of > > > > > this > > > > > > > > FLIP. > > > > > > > > >> > > > > > > > > > >> > @till @stephan, please take another look see if there > are > > > any > > > > > > other > > > > > > > > >> > concerns. > > > > > > > > >> > > > > > > > > > >> > Thank you~ > > > > > > > > >> > > > > > > > > > >> > Xintong Song > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > [1] > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > >> > > > > > > > > > >> > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song < > > > > > > [hidden email] > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > >> > > Sorry for the late response. > > > > > > > > >> > > > > > > > > > > >> > > - Regarding the `TaskExecutorSpecifics` naming, let's > > > > discuss > > > > > > the > > > > > > > > >> detail > > > > > > > > >> > > in PR. > > > > > > > > >> > > - Regarding passing parameters into the > `TaskExecutor`, > > +1 > > > > for > > > > > > > using > > > > > > > > >> > > dynamic configuration at the moment, given that there > > are > > > > more > > > > > > > > >> questions > > > > > > > > >> > to > > > > > > > > >> > > be discussed to have a general framework for > overwriting > > > > > > > > >> configurations > > > > > > > > >> > > with ENV variables. > > > > > > > > >> > > - Regarding memory reservation, I double checked with > Yu > > > and > > > > > he > > > > > > > will > > > > > > > > >> take > > > > > > > > >> > > care of it. > > > > > > > > >> > > > > > > > > > > >> > > Thank you~ > > > > > > > > >> > > > > > > > > > > >> > > Xintong Song > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > >> > > > > > > > > > > >> > >> What I forgot to add is that we could tackle > specifying > > > the > > > > > > > > >> > configuration > > > > > > > > >> > >> fully in an incremental way and that the full > > > specification > > > > > > > should > > > > > > > > be > > > > > > > > >> > the > > > > > > > > >> > >> desired end state. > > > > > > > > >> > >> > > > > > > > > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann < > > > > > > > > [hidden email]> > > > > > > > > >> > >> wrote: > > > > > > > > >> > >> > > > > > > > > >> > >> > I think our goal should be that the configuration > is > > > > fully > > > > > > > > >> specified > > > > > > > > >> > >> when > > > > > > > > >> > >> > the process is started. By considering the internal > > > > > > calculation > > > > > > > > >> step > > > > > > > > >> > to > > > > > > > > >> > >> be > > > > > > > > >> > >> > rather validate existing values and calculate > missing > > > > ones, > > > > > > > these > > > > > > > > >> two > > > > > > > > >> > >> > proposal shouldn't even conflict (given > determinism). > > > > > > > > >> > >> > > > > > > > > > >> > >> > Since we don't want to change an existing > > > > flink-conf.yaml, > > > > > > > > >> specifying > > > > > > > > >> > >> the > > > > > > > > >> > >> > full configuration would require to pass in the > > options > > > > > > > > >> differently. > > > > > > > > >> > >> > > > > > > > > > >> > >> > One way could be the ENV variables approach. The > > reason > > > > why > > > > > > I'm > > > > > > > > >> trying > > > > > > > > >> > >> to > > > > > > > > >> > >> > exclude this feature from the FLIP is that I > believe > > it > > > > > > needs a > > > > > > > > bit > > > > > > > > >> > more > > > > > > > > >> > >> > discussion. Just some questions which come to my > > mind: > > > > What > > > > > > > would > > > > > > > > >> be > > > > > > > > >> > the > > > > > > > > >> > >> > exact format (FLINK_KEY_NAME)? Would we support a > dot > > > > > > separator > > > > > > > > >> which > > > > > > > > >> > is > > > > > > > > >> > >> > supported by some systems (FLINK.KEY.NAME)? If we > > > accept > > > > > the > > > > > > > dot > > > > > > > > >> > >> > separator what would be the order of precedence if > > > there > > > > > are > > > > > > > two > > > > > > > > >> ENV > > > > > > > > >> > >> > variables defined (FLINK_KEY_NAME and > FLINK.KEY.NAME > > )? > > > > > What > > > > > > is > > > > > > > > the > > > > > > > > >> > >> > precedence of env variable vs. dynamic > configuration > > > > value > > > > > > > > >> specified > > > > > > > > >> > >> via -D? > > > > > > > > >> > >> > > > > > > > > > >> > >> > Another approach could be to pass in the dynamic > > > > > > configuration > > > > > > > > >> values > > > > > > > > >> > >> via > > > > > > > > >> > >> > `-Dkey=value` to the Flink process. For that we > don't > > > > have > > > > > to > > > > > > > > >> change > > > > > > > > >> > >> > anything because the functionality already exists. > > > > > > > > >> > >> > > > > > > > > > >> > >> > Cheers, > > > > > > > > >> > >> > Till > > > > > > > > >> > >> > > > > > > > > > >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan Ewen < > > > > > > > [hidden email]> > > > > > > > > >> > wrote: > > > > > > > > >> > >> > > > > > > > > > >> > >> >> I see. Under the assumption of strict determinism > > that > > > > > > should > > > > > > > > >> work. > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> The original proposal had this point "don't > compute > > > > inside > > > > > > the > > > > > > > > TM, > > > > > > > > >> > >> compute > > > > > > > > >> > >> >> outside and supply a full config", because that > > > sounded > > > > > more > > > > > > > > >> > intuitive. > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann < > > > > > > > > >> [hidden email] > > > > > > > > >> > > > > > > > > > > >> > >> >> wrote: > > > > > > > > >> > >> >> > > > > > > > > >> > >> >> > My understanding was that before starting the > > Flink > > > > > > process > > > > > > > we > > > > > > > > >> > call a > > > > > > > > >> > >> >> > utility which calculates these values. I assume > > that > > > > > this > > > > > > > > >> utility > > > > > > > > >> > >> will > > > > > > > > >> > >> >> do > > > > > > > > >> > >> >> > the calculation based on a set of configured > > values > > > > > > (process > > > > > > > > >> > memory, > > > > > > > > >> > >> >> flink > > > > > > > > >> > >> >> > memory, network memory etc.). Assuming that > these > > > > values > > > > > > > don't > > > > > > > > >> > differ > > > > > > > > >> > >> >> from > > > > > > > > >> > >> >> > the values with which the JVM is started, it > > should > > > be > > > > > > > > possible > > > > > > > > >> to > > > > > > > > >> > >> >> > recompute them in the Flink process in order to > > set > > > > the > > > > > > > > values. > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen < > > > > > > > > [hidden email] > > > > > > > > >> > > > > > > > > > >> > >> wrote: > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > When computing the values in the JVM process > > after > > > > it > > > > > > > > started, > > > > > > > > >> > how > > > > > > > > >> > >> >> would > > > > > > > > >> > >> >> > > you deal with values like Max Direct Memory, > > > > Metaspace > > > > > > > size. > > > > > > > > >> > native > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > reservation (reduce heap size), etc? All the > > > values > > > > > that > > > > > > > are > > > > > > > > >> > >> >> parameters > > > > > > > > >> > >> >> > to > > > > > > > > >> > >> >> > > the JVM process and that need to be supplied > at > > > > > process > > > > > > > > >> startup? > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till Rohrmann > < > > > > > > > > >> > >> [hidden email]> > > > > > > > > >> > >> >> > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > Thanks for the clarification. I have some > more > > > > > > comments: > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > - I would actually split the logic to > compute > > > the > > > > > > > process > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > > requirements and storing the values into two > > > > things. > > > > > > > E.g. > > > > > > > > >> one > > > > > > > > >> > >> could > > > > > > > > >> > >> >> > name > > > > > > > > >> > >> >> > > > the former TaskExecutorProcessUtility and > the > > > > > latter > > > > > > > > >> > >> >> > > > TaskExecutorProcessMemory. But we can > discuss > > > this > > > > > on > > > > > > > the > > > > > > > > PR > > > > > > > > >> > >> since > > > > > > > > >> > >> >> it's > > > > > > > > >> > >> >> > > > just a naming detail. > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > - Generally, I'm not opposed to making > > > > configuration > > > > > > > > values > > > > > > > > >> > >> >> overridable > > > > > > > > >> > >> >> > > by > > > > > > > > >> > >> >> > > > ENV variables. I think this is a very good > > idea > > > > and > > > > > > > makes > > > > > > > > >> the > > > > > > > > >> > >> >> > > > configurability of Flink processes easier. > > > > However, > > > > > I > > > > > > > > think > > > > > > > > >> > that > > > > > > > > >> > >> >> adding > > > > > > > > >> > >> >> > > > this functionality should not be part of > this > > > FLIP > > > > > > > because > > > > > > > > >> it > > > > > > > > >> > >> would > > > > > > > > >> > >> >> > > simply > > > > > > > > >> > >> >> > > > widen the scope unnecessarily. > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > The reasons why I believe it is unnecessary > > are > > > > the > > > > > > > > >> following: > > > > > > > > >> > >> For > > > > > > > > >> > >> >> Yarn > > > > > > > > >> > >> >> > > we > > > > > > > > >> > >> >> > > > already create write a flink-conf.yaml which > > > could > > > > > be > > > > > > > > >> populated > > > > > > > > >> > >> with > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > memory settings. For the other processes it > > > should > > > > > not > > > > > > > > make > > > > > > > > >> a > > > > > > > > >> > >> >> > difference > > > > > > > > >> > >> >> > > > whether the loaded Configuration is > populated > > > with > > > > > the > > > > > > > > >> memory > > > > > > > > >> > >> >> settings > > > > > > > > >> > >> >> > > from > > > > > > > > >> > >> >> > > > ENV variables or by using > > > > TaskExecutorProcessUtility > > > > > > to > > > > > > > > >> compute > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > missing > > > > > > > > >> > >> >> > > > values from the loaded configuration. If the > > > > latter > > > > > > > would > > > > > > > > >> not > > > > > > > > >> > be > > > > > > > > >> > >> >> > possible > > > > > > > > >> > >> >> > > > (wrong or missing configuration values), > then > > we > > > > > > should > > > > > > > > not > > > > > > > > >> > have > > > > > > > > >> > >> >> been > > > > > > > > >> > >> >> > > able > > > > > > > > >> > >> >> > > > to actually start the process in the first > > > place. > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > - Concerning the memory reservation: I agree > > > with > > > > > you > > > > > > > that > > > > > > > > >> we > > > > > > > > >> > >> need > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > memory reservation functionality to make > > > streaming > > > > > > jobs > > > > > > > > work > > > > > > > > >> > with > > > > > > > > >> > >> >> > > "managed" > > > > > > > > >> > >> >> > > > memory. However, w/o this functionality the > > > whole > > > > > Flip > > > > > > > > would > > > > > > > > >> > >> already > > > > > > > > >> > >> >> > > bring > > > > > > > > >> > >> >> > > > a good amount of improvements to our users > > when > > > > > > running > > > > > > > > >> batch > > > > > > > > >> > >> jobs. > > > > > > > > >> > >> >> > > > Moreover, by keeping the scope smaller we > can > > > > > complete > > > > > > > the > > > > > > > > >> FLIP > > > > > > > > >> > >> >> faster. > > > > > > > > >> > >> >> > > > Hence, I would propose to address the memory > > > > > > reservation > > > > > > > > >> > >> >> functionality > > > > > > > > >> > >> >> > > as a > > > > > > > > >> > >> >> > > > follow up FLIP (which Yu is working on if > I'm > > > not > > > > > > > > mistaken). > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > Cheers, > > > > > > > > >> > >> >> > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang Wang < > > > > > > > > >> > >> [hidden email]> > > > > > > > > >> > >> >> > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > Just add my 2 cents. > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > Using environment variables to override > the > > > > > > > > configuration > > > > > > > > >> for > > > > > > > > >> > >> >> > different > > > > > > > > >> > >> >> > > > > taskmanagers is better. > > > > > > > > >> > >> >> > > > > We do not need to generate dedicated > > > > > flink-conf.yaml > > > > > > > for > > > > > > > > >> all > > > > > > > > >> > >> >> > > > taskmanagers. > > > > > > > > >> > >> >> > > > > A common flink-conf.yam and different > > > > environment > > > > > > > > >> variables > > > > > > > > >> > are > > > > > > > > >> > >> >> > enough. > > > > > > > > >> > >> >> > > > > By reducing the distributed cached files, > it > > > > could > > > > > > > make > > > > > > > > >> > >> launching > > > > > > > > >> > >> >> a > > > > > > > > >> > >> >> > > > > taskmanager faster. > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > Stephan gives a good suggestion that we > > could > > > > move > > > > > > the > > > > > > > > >> logic > > > > > > > > >> > >> into > > > > > > > > >> > >> >> > > > > "GlobalConfiguration.loadConfig()" method. > > > > > > > > >> > >> >> > > > > Maybe the client could also benefit from > > this. > > > > > > > Different > > > > > > > > >> > users > > > > > > > > >> > >> do > > > > > > > > >> > >> >> not > > > > > > > > >> > >> >> > > > have > > > > > > > > >> > >> >> > > > > to export FLINK_CONF_DIR to update few > > config > > > > > > options. > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > Best, > > > > > > > > >> > >> >> > > > > Yang > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > Stephan Ewen <[hidden email]> > > 于2019年8月28日周三 > > > > > > > 上午1:21写道： > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > One note on the Environment Variables > and > > > > > > > > Configuration > > > > > > > > >> > >> >> discussion. > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > My understanding is that passed ENV > > > variables > > > > > are > > > > > > > > added > > > > > > > > >> to > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > > > > configuration in the > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > >> > >> method > > > > > > > > >> > >> >> (or > > > > > > > > >> > >> >> > > > > > similar). > > > > > > > > >> > >> >> > > > > > For all the code inside Flink, it looks > > like > > > > the > > > > > > > data > > > > > > > > >> was > > > > > > > > >> > in > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > config > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > start with, just that the scripts that > > > compute > > > > > the > > > > > > > > >> > variables > > > > > > > > >> > >> can > > > > > > > > >> > >> >> > pass > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > values to the process without actually > > > needing > > > > > to > > > > > > > > write > > > > > > > > >> a > > > > > > > > >> > >> file. > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > For example the > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > >> method > > > > > > > > >> > >> would > > > > > > > > >> > >> >> > take > > > > > > > > >> > >> >> > > > any > > > > > > > > >> > >> >> > > > > > ENV variable prefixed with "flink" and > add > > > it > > > > > as a > > > > > > > > >> config > > > > > > > > >> > >> key. > > > > > > > > >> > >> >> > > > > > "flink_taskmanager_memory_size=2g" would > > > > become > > > > > > > > >> > >> >> > > > "taskmanager.memory.size: > > > > > > > > >> > >> >> > > > > > 2g". > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM Xintong > > > Song < > > > > > > > > >> > >> >> > [hidden email]> > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > Thanks for the comments, Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > I've also seen your comments on the > wiki > > > > page, > > > > > > but > > > > > > > > >> let's > > > > > > > > >> > >> keep > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > > > > discussion here. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding 'TaskExecutorSpecifics', > how > > > do > > > > > you > > > > > > > > think > > > > > > > > >> > about > > > > > > > > >> > >> >> > naming > > > > > > > > >> > >> >> > > it > > > > > > > > >> > >> >> > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > > > >> > >> >> > > > > > > - Regarding passing memory > > configurations > > > > into > > > > > > > task > > > > > > > > >> > >> executors, > > > > > > > > >> > >> >> > I'm > > > > > > > > >> > >> >> > > in > > > > > > > > >> > >> >> > > > > > favor > > > > > > > > >> > >> >> > > > > > > of do it via environment variables > > rather > > > > than > > > > > > > > >> > >> configurations, > > > > > > > > >> > >> >> > with > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > following two reasons. > > > > > > > > >> > >> >> > > > > > > - It is easier to keep the memory > > > options > > > > > once > > > > > > > > >> > calculate > > > > > > > > >> > >> >> not to > > > > > > > > >> > >> >> > > be > > > > > > > > >> > >> >> > > > > > > changed with environment variables > > rather > > > > than > > > > > > > > >> > >> configurations. > > > > > > > > >> > >> >> > > > > > > - I'm not sure whether we should > write > > > the > > > > > > > > >> > configuration > > > > > > > > >> > >> in > > > > > > > > >> > >> >> > > startup > > > > > > > > >> > >> >> > > > > > > scripts. Writing changes into the > > > > > configuration > > > > > > > > files > > > > > > > > >> > when > > > > > > > > >> > >> >> > running > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > startup scripts does not sounds right > to > > > me. > > > > > Or > > > > > > we > > > > > > > > >> could > > > > > > > > >> > >> make > > > > > > > > >> > >> >> a > > > > > > > > >> > >> >> > > copy > > > > > > > > >> > >> >> > > > of > > > > > > > > >> > >> >> > > > > > > configuration files per flink cluster, > > and > > > > > make > > > > > > > the > > > > > > > > >> task > > > > > > > > >> > >> >> executor > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > load > > > > > > > > >> > >> >> > > > > > > from the copy, and clean up the copy > > after > > > > the > > > > > > > > >> cluster is > > > > > > > > >> > >> >> > shutdown, > > > > > > > > >> > >> >> > > > > which > > > > > > > > >> > >> >> > > > > > > is complicated. (I think this is also > > what > > > > > > Stephan > > > > > > > > >> means > > > > > > > > >> > in > > > > > > > > >> > >> >> his > > > > > > > > >> > >> >> > > > comment > > > > > > > > >> > >> >> > > > > > on > > > > > > > > >> > >> >> > > > > > > the wiki page?) > > > > > > > > >> > >> >> > > > > > > - Regarding reserving memory, I think > > this > > > > > > change > > > > > > > > >> should > > > > > > > > >> > be > > > > > > > > >> > >> >> > > included > > > > > > > > >> > >> >> > > > in > > > > > > > > >> > >> >> > > > > > > this FLIP. I think a big part of > > > motivations > > > > > of > > > > > > > this > > > > > > > > >> FLIP > > > > > > > > >> > >> is > > > > > > > > >> > >> >> to > > > > > > > > >> > >> >> > > unify > > > > > > > > >> > >> >> > > > > > > memory configuration for streaming / > > batch > > > > and > > > > > > > make > > > > > > > > it > > > > > > > > >> > easy > > > > > > > > >> > >> >> for > > > > > > > > >> > >> >> > > > > > configuring > > > > > > > > >> > >> >> > > > > > > rocksdb memory. If we don't support > > memory > > > > > > > > >> reservation, > > > > > > > > >> > >> then > > > > > > > > >> > >> >> > > > streaming > > > > > > > > >> > >> >> > > > > > jobs > > > > > > > > >> > >> >> > > > > > > cannot use managed memory (neither > > on-heap > > > > or > > > > > > > > >> off-heap), > > > > > > > > >> > >> which > > > > > > > > >> > >> >> > > makes > > > > > > > > >> > >> >> > > > > this > > > > > > > > >> > >> >> > > > > > > FLIP incomplete. > > > > > > > > >> > >> >> > > > > > > - Regarding network memory, I think > you > > > are > > > > > > > right. I > > > > > > > > >> > think > > > > > > > > >> > >> we > > > > > > > > >> > >> >> > > > probably > > > > > > > > >> > >> >> > > > > > > don't need to change network stack > from > > > > using > > > > > > > direct > > > > > > > > >> > >> memory to > > > > > > > > >> > >> >> > > using > > > > > > > > >> > >> >> > > > > > unsafe > > > > > > > > >> > >> >> > > > > > > native memory. Network memory size is > > > > > > > deterministic, > > > > > > > > >> > >> cannot be > > > > > > > > >> > >> >> > > > reserved > > > > > > > > >> > >> >> > > > > > as > > > > > > > > >> > >> >> > > > > > > managed memory does, and cannot be > > > > overused. I > > > > > > > think > > > > > > > > >> it > > > > > > > > >> > >> also > > > > > > > > >> > >> >> > works > > > > > > > > >> > >> >> > > if > > > > > > > > >> > >> >> > > > > we > > > > > > > > >> > >> >> > > > > > > simply keep using direct memory for > > > network > > > > > and > > > > > > > > >> include > > > > > > > > >> > it > > > > > > > > >> > >> in > > > > > > > > >> > >> >> jvm > > > > > > > > >> > >> >> > > max > > > > > > > > >> > >> >> > > > > > > direct memory size. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till > > > > Rohrmann > > > > > < > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > Hi Xintong, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > thanks for addressing the comments > and > > > > > adding > > > > > > a > > > > > > > > more > > > > > > > > >> > >> >> detailed > > > > > > > > >> > >> >> > > > > > > > implementation plan. I have a couple > > of > > > > > > comments > > > > > > > > >> > >> concerning > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > > > > > implementation plan: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > - The name `TaskExecutorSpecifics` > is > > > not > > > > > > really > > > > > > > > >> > >> >> descriptive. > > > > > > > > >> > >> >> > > > > Choosing > > > > > > > > >> > >> >> > > > > > a > > > > > > > > >> > >> >> > > > > > > > different name could help here. > > > > > > > > >> > >> >> > > > > > > > - I'm not sure whether I would pass > > the > > > > > memory > > > > > > > > >> > >> >> configuration to > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > > > TaskExecutor via environment > > variables. > > > I > > > > > > think > > > > > > > it > > > > > > > > >> > would > > > > > > > > >> > >> be > > > > > > > > >> > >> >> > > better > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > write > > > > > > > > >> > >> >> > > > > > > > it into the configuration one uses > to > > > > start > > > > > > the > > > > > > > TM > > > > > > > > >> > >> process. > > > > > > > > >> > >> >> > > > > > > > - If possible, I would exclude the > > > memory > > > > > > > > >> reservation > > > > > > > > >> > >> from > > > > > > > > >> > >> >> this > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > >> > >> >> > > > > > and > > > > > > > > >> > >> >> > > > > > > > add this as part of a dedicated > FLIP. > > > > > > > > >> > >> >> > > > > > > > - If possible, then I would exclude > > > > changes > > > > > to > > > > > > > the > > > > > > > > >> > >> network > > > > > > > > >> > >> >> > stack > > > > > > > > >> > >> >> > > > from > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > >> > >> >> > > > > > > > FLIP. Maybe we can simply say that > the > > > > > direct > > > > > > > > memory > > > > > > > > >> > >> needed > > > > > > > > >> > >> >> by > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > > network > > > > > > > > >> > >> >> > > > > > > > stack is the framework direct memory > > > > > > > requirement. > > > > > > > > >> > >> Changing > > > > > > > > >> > >> >> how > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > is allocated can happen in a second > > > step. > > > > > This > > > > > > > > would > > > > > > > > >> > keep > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > scope > > > > > > > > >> > >> >> > > > > of > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > >> > >> >> > > > > > > > FLIP smaller. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM > > Xintong > > > > > Song < > > > > > > > > >> > >> >> > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Hi everyone, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > I just updated the FLIP document > on > > > wiki > > > > > > [1], > > > > > > > > with > > > > > > > > >> > the > > > > > > > > >> > >> >> > > following > > > > > > > > >> > >> >> > > > > > > changes. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > - Removed open question > regarding > > > > > > > > MemorySegment > > > > > > > > >> > >> >> > allocation. > > > > > > > > >> > >> >> > > As > > > > > > > > >> > >> >> > > > > > > > > discussed, we exclude this > topic > > > from > > > > > the > > > > > > > > >> scope of > > > > > > > > >> > >> this > > > > > > > > >> > >> >> > > FLIP. > > > > > > > > >> > >> >> > > > > > > > > - Updated content about JVM > > direct > > > > > memory > > > > > > > > >> > parameter > > > > > > > > >> > >> >> > > according > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > recent > > > > > > > > >> > >> >> > > > > > > > > discussions, and moved the > other > > > > > options > > > > > > to > > > > > > > > >> > >> "Rejected > > > > > > > > >> > >> >> > > > > > Alternatives" > > > > > > > > >> > >> >> > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > moment. > > > > > > > > >> > >> >> > > > > > > > > - Added implementation steps. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > [1] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 PM > > > Stephan > > > > > > Ewen < > > > > > > > > >> > >> >> > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong: Concerning "wait for > > > memory > > > > > > users > > > > > > > > >> before > > > > > > > > >> > >> task > > > > > > > > >> > >> >> > > dispose > > > > > > > > >> > >> >> > > > > and > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > release": I agree, that's how it > > > > should > > > > > > be. > > > > > > > > >> Let's > > > > > > > > >> > >> try it > > > > > > > > >> > >> >> > out. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong @Jingsong: Concerning " > > JVM > > > > > does > > > > > > > not > > > > > > > > >> wait > > > > > > > > >> > >> for > > > > > > > > >> > >> >> GC > > > > > > > > >> > >> >> > > when > > > > > > > > >> > >> >> > > > > > > > allocating > > > > > > > > >> > >> >> > > > > > > > > > direct memory buffer": There > seems > > > to > > > > be > > > > > > > > pretty > > > > > > > > >> > >> >> elaborate > > > > > > > > >> > >> >> > > logic > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > free > > > > > > > > >> > >> >> > > > > > > > > > buffers when allocating new > ones. > > > See > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Till: Maybe. If we assume that > > the > > > > JVM > > > > > > > > default > > > > > > > > >> > works > > > > > > > > >> > >> >> (like > > > > > > > > >> > >> >> > > > going > > > > > > > > >> > >> >> > > > > > > with > > > > > > > > >> > >> >> > > > > > > > > > option 2 and not setting > > > > > > > > >> "-XX:MaxDirectMemorySize" > > > > > > > > >> > at > > > > > > > > >> > >> >> all), > > > > > > > > >> > >> >> > > > then > > > > > > > > >> > >> >> > > > > I > > > > > > > > >> > >> >> > > > > > > > think > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > > should be okay to set > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > >> to > > > > > > > > >> > >> >> > > > > > > > > > "off_heap_managed_memory + > > > > > direct_memory" > > > > > > > even > > > > > > > > >> if > > > > > > > > >> > we > > > > > > > > >> > >> use > > > > > > > > >> > >> >> > > > RocksDB. > > > > > > > > >> > >> >> > > > > > > That > > > > > > > > >> > >> >> > > > > > > > > is a > > > > > > > > >> > >> >> > > > > > > > > > big if, though, I honestly have > no > > > > idea > > > > > :D > > > > > > > > >> Would be > > > > > > > > >> > >> >> good to > > > > > > > > >> > >> >> > > > > > > understand > > > > > > > > >> > >> >> > > > > > > > > > this, though, because this would > > > > affect > > > > > > > option > > > > > > > > >> (2) > > > > > > > > >> > >> and > > > > > > > > >> > >> >> > option > > > > > > > > >> > >> >> > > > > > (1.2). > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 PM > > > > Xintong > > > > > > > Song < > > > > > > > > >> > >> >> > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thanks for the inputs, > Jingsong. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Let me try to summarize your > > > points. > > > > > > > Please > > > > > > > > >> > correct > > > > > > > > >> > >> >> me if > > > > > > > > >> > >> >> > > I'm > > > > > > > > >> > >> >> > > > > > > wrong. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - Memory consumers should > > > always > > > > > > avoid > > > > > > > > >> > returning > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > > > > segments > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > memory manager while there > > are > > > > > still > > > > > > > > >> > un-cleaned > > > > > > > > >> > >> >> > > > structures / > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > may use the memory. > > Otherwise, > > > it > > > > > > would > > > > > > > > >> cause > > > > > > > > >> > >> >> serious > > > > > > > > >> > >> >> > > > > problems > > > > > > > > >> > >> >> > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > >> > >> >> > > > > > > > > > > multiple consumers trying > to > > > use > > > > > the > > > > > > > same > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > segment. > > > > > > > > >> > >> >> > > > > > > > > > > - JVM does not wait for GC > > when > > > > > > > > allocating > > > > > > > > >> > >> direct > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > > > > buffer. > > > > > > > > >> > >> >> > > > > > > > > > > Therefore even we set > proper > > > max > > > > > > direct > > > > > > > > >> memory > > > > > > > > >> > >> size > > > > > > > > >> > >> >> > > limit, > > > > > > > > >> > >> >> > > > > we > > > > > > > > >> > >> >> > > > > > > may > > > > > > > > >> > >> >> > > > > > > > > > still > > > > > > > > >> > >> >> > > > > > > > > > > encounter direct memory oom > > if > > > > the > > > > > GC > > > > > > > > >> cleaning > > > > > > > > >> > >> >> memory > > > > > > > > >> > >> >> > > > slower > > > > > > > > >> > >> >> > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > direct memory allocation. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Am I understanding this > > correctly? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at 4:21 > PM > > > > > > > JingsongLee > > > > > > > > < > > > > > > > > >> > >> >> > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > .invalid> > > > > > > > > >> > >> >> > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Hi stephan: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About option 2: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > if additional threads not > > > cleanly > > > > > shut > > > > > > > > down > > > > > > > > >> > >> before > > > > > > > > >> > >> >> we > > > > > > > > >> > >> >> > can > > > > > > > > >> > >> >> > > > > exit > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > task: > > > > > > > > >> > >> >> > > > > > > > > > > > In the current case of > memory > > > > reuse, > > > > > > it > > > > > > > > has > > > > > > > > >> > >> freed up > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > > memory > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > > > > uses. If this memory is > used > > by > > > > > other > > > > > > > > tasks > > > > > > > > >> > and > > > > > > > > >> > >> >> > > > asynchronous > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > >> > >> >> > > > > > > > > > > > of exited task may still be > > > > > writing, > > > > > > > > there > > > > > > > > >> > will > > > > > > > > >> > >> be > > > > > > > > >> > >> >> > > > > concurrent > > > > > > > > >> > >> >> > > > > > > > > security > > > > > > > > >> > >> >> > > > > > > > > > > > problems, and even lead to > > > errors > > > > > in > > > > > > > user > > > > > > > > >> > >> computing > > > > > > > > >> > >> >> > > > results. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > So I think this is a serious > > and > > > > > > > > intolerable > > > > > > > > >> > >> bug, No > > > > > > > > >> > >> >> > > matter > > > > > > > > >> > >> >> > > > > > what > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > option is, it should be > > > avoided. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About direct memory cleaned > by > > > GC: > > > > > > > > >> > >> >> > > > > > > > > > > > I don't think it is a good > > idea, > > > > > I've > > > > > > > > >> > >> encountered so > > > > > > > > >> > >> >> > many > > > > > > > > >> > >> >> > > > > > > > situations > > > > > > > > >> > >> >> > > > > > > > > > > > that it's too late for GC > to > > > > cause > > > > > > > > >> > DirectMemory > > > > > > > > >> > >> >> OOM. > > > > > > > > >> > >> >> > > > Release > > > > > > > > >> > >> >> > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > allocate DirectMemory > depend > > on > > > > the > > > > > > > type > > > > > > > > of > > > > > > > > >> > user > > > > > > > > >> > >> >> job, > > > > > > > > >> > >> >> > > > which > > > > > > > > >> > >> >> > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > often beyond our control. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Best, > > > > > > > > >> > >> >> > > > > > > > > > > > Jingsong Lee > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > >> > > > > ------------------------------------------------------------------ > > > > > > > > >> > >> >> > > > > > > > > > > > From:Stephan Ewen < > > > > [hidden email] > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Send Time:2019年8月19日(星期一) > > 15:56 > > > > > > > > >> > >> >> > > > > > > > > > > > To:dev < > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] > FLIP-49: > > > > > Unified > > > > > > > > >> Memory > > > > > > > > >> > >> >> > > Configuration > > > > > > > > >> > >> >> > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > TaskExecutors > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > My main concern with option > 2 > > > > > > (manually > > > > > > > > >> release > > > > > > > > >> > >> >> memory) > > > > > > > > >> > >> >> > > is > > > > > > > > >> > >> >> > > > > that > > > > > > > > >> > >> >> > > > > > > > > > segfaults > > > > > > > > >> > >> >> > > > > > > > > > > > in the JVM send off all > sorts > > of > > > > > > alarms > > > > > > > on > > > > > > > > >> user > > > > > > > > >> > >> >> ends. > > > > > > > > >> > >> >> > So > > > > > > > > >> > >> >> > > we > > > > > > > > >> > >> >> > > > > > need > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > guarantee that this never > > > happens. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > The trickyness is in tasks > > that > > > > uses > > > > > > > data > > > > > > > > >> > >> >> structures / > > > > > > > > >> > >> >> > > > > > algorithms > > > > > > > > >> > >> >> > > > > > > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > additional threads, like > hash > > > > table > > > > > > > > >> spill/read > > > > > > > > >> > >> and > > > > > > > > >> > >> >> > > sorting > > > > > > > > >> > >> >> > > > > > > threads. > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > >> > >> >> > > > > > > > > > > > to ensure that these cleanly > > > shut > > > > > down > > > > > > > > >> before > > > > > > > > >> > we > > > > > > > > >> > >> can > > > > > > > > >> > >> >> > exit > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > task. > > > > > > > > >> > >> >> > > > > > > > > > > > I am not sure that we have > > that > > > > > > > guaranteed > > > > > > > > >> > >> already, > > > > > > > > >> > >> >> > > that's > > > > > > > > >> > >> >> > > > > why > > > > > > > > >> > >> >> > > > > > > > option > > > > > > > > >> > >> >> > > > > > > > > > 1.1 > > > > > > > > >> > >> >> > > > > > > > > > > > seemed simpler to me. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 at 3:42 > > PM > > > > > > Xintong > > > > > > > > >> Song < > > > > > > > > >> > >> >> > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thanks for the comments, > > > > Stephan. > > > > > > > > >> Summarized > > > > > > > > >> > in > > > > > > > > >> > >> >> this > > > > > > > > >> > >> >> > > way > > > > > > > > >> > >> >> > > > > > really > > > > > > > > >> > >> >> > > > > > > > > makes > > > > > > > > >> > >> >> > > > > > > > > > > > > things easier to > understand. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'm in favor of option 2, > at > > > > least > > > > > > for > > > > > > > > the > > > > > > > > >> > >> >> moment. I > > > > > > > > >> > >> >> > > > think > > > > > > > > >> > >> >> > > > > it > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > difficult to keep it > > segfault > > > > safe > > > > > > for > > > > > > > > >> memory > > > > > > > > >> > >> >> > manager, > > > > > > > > >> > >> >> > > as > > > > > > > > >> > >> >> > > > > > long > > > > > > > > >> > >> >> > > > > > > as > > > > > > > > >> > >> >> > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > always > > > > > > > > >> > >> >> > > > > > > > > > > > > de-allocate the memory > > segment > > > > > when > > > > > > it > > > > > > > > is > > > > > > > > >> > >> released > > > > > > > > >> > >> >> > from > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > consumers. Only if the > > memory > > > > > > consumer > > > > > > > > >> > continue > > > > > > > > >> > >> >> using > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > > buffer > > > > > > > > >> > >> >> > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > segment after releasing > it, > > in > > > > > which > > > > > > > > case > > > > > > > > >> we > > > > > > > > >> > do > > > > > > > > >> > >> >> want > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > job > > > > > > > > >> > >> >> > > > > > to > > > > > > > > >> > >> >> > > > > > > > > fail > > > > > > > > >> > >> >> > > > > > > > > > so > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > detect the memory leak > > early. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.2, I don't > > think > > > > this > > > > > > is > > > > > > > a > > > > > > > > >> good > > > > > > > > >> > >> idea. > > > > > > > > >> > >> >> > Not > > > > > > > > >> > >> >> > > > only > > > > > > > > >> > >> >> > > > > > > > because > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > assumption (regular GC is > > > enough > > > > > to > > > > > > > > clean > > > > > > > > >> > >> direct > > > > > > > > >> > >> >> > > buffers) > > > > > > > > >> > >> >> > > > > may > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > always > > > > > > > > >> > >> >> > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > true, but also it makes > > harder > > > > for > > > > > > > > finding > > > > > > > > >> > >> >> problems > > > > > > > > >> > >> >> > in > > > > > > > > >> > >> >> > > > > cases > > > > > > > > >> > >> >> > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > overuse. E.g., user > > configured > > > > > some > > > > > > > > direct > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> for > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > libraries. > > > > > > > > >> > >> >> > > > > > > > > > > > > If the library actually > use > > > more > > > > > > > direct > > > > > > > > >> > memory > > > > > > > > >> > >> >> then > > > > > > > > >> > >> >> > > > > > configured, > > > > > > > > >> > >> >> > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > > cannot be cleaned by GC > > > because > > > > > they > > > > > > > are > > > > > > > > >> > still > > > > > > > > >> > >> in > > > > > > > > >> > >> >> > use, > > > > > > > > >> > >> >> > > > may > > > > > > > > >> > >> >> > > > > > lead > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > overuse > > > > > > > > >> > >> >> > > > > > > > > > > > > of the total container > > memory. > > > > In > > > > > > that > > > > > > > > >> case, > > > > > > > > >> > >> if it > > > > > > > > >> > >> >> > > didn't > > > > > > > > >> > >> >> > > > > > touch > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > default max direct memory > > > limit, > > > > > we > > > > > > > > cannot > > > > > > > > >> > get > > > > > > > > >> > >> a > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > OOM > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > > > > > will become super hard to > > > > > understand > > > > > > > > which > > > > > > > > >> > >> part of > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > >> > >> >> > > > > > > > > > > > > to be updated. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.1, it has the > > > > similar > > > > > > > > >> problem as > > > > > > > > >> > >> >> 1.2, if > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > > > exceeded > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > memory does not reach the > > max > > > > > direct > > > > > > > > >> memory > > > > > > > > >> > >> limit > > > > > > > > >> > >> >> > > > specified > > > > > > > > >> > >> >> > > > > > by > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > dedicated parameter. I > think > > > it > > > > is > > > > > > > > >> slightly > > > > > > > > >> > >> better > > > > > > > > >> > >> >> > than > > > > > > > > >> > >> >> > > > > 1.2, > > > > > > > > >> > >> >> > > > > > > only > > > > > > > > >> > >> >> > > > > > > > > > > because > > > > > > > > >> > >> >> > > > > > > > > > > > > we can tune the parameter. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 at > 2:53 > > > PM > > > > > > > Stephan > > > > > > > > >> Ewen > > > > > > > > >> > < > > > > > > > > >> > >> >> > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > About the > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > >> > >> discussion, > > > > > > > > >> > >> >> > maybe > > > > > > > > >> > >> >> > > > let > > > > > > > > >> > >> >> > > > > > me > > > > > > > > >> > >> >> > > > > > > > > > > summarize > > > > > > > > >> > >> >> > > > > > > > > > > > > it a > > > > > > > > >> > >> >> > > > > > > > > > > > > > bit differently: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > We have the following > two > > > > > options: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (1) We let > MemorySegments > > be > > > > > > > > >> de-allocated > > > > > > > > >> > by > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > GC. > > > > > > > > >> > >> >> > > > That > > > > > > > > >> > >> >> > > > > > > makes > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > > > > > segfault > > > > > > > > >> > >> >> > > > > > > > > > > > > > safe. But then we need a > > way > > > > to > > > > > > > > trigger > > > > > > > > >> GC > > > > > > > > >> > in > > > > > > > > >> > >> >> case > > > > > > > > >> > >> >> > > > > > > > de-allocation > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > re-allocation of a bunch > > of > > > > > > segments > > > > > > > > >> > happens > > > > > > > > >> > >> >> > quickly, > > > > > > > > >> > >> >> > > > > which > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > often > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > case during batch > > scheduling > > > > or > > > > > > task > > > > > > > > >> > restart. > > > > > > > > >> > >> >> > > > > > > > > > > > > > - The > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > >> (option > > > > > > > > >> > >> 1.1) > > > > > > > > >> > >> >> is > > > > > > > > >> > >> >> > one > > > > > > > > >> > >> >> > > > way > > > > > > > > >> > >> >> > > > > > to > > > > > > > > >> > >> >> > > > > > > do > > > > > > > > >> > >> >> > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Another way could be > > to > > > > > have a > > > > > > > > >> > dedicated > > > > > > > > >> > >> >> > > > bookkeeping > > > > > > > > >> > >> >> > > > > in > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > MemoryManager (option > > 1.2), > > > so > > > > > > that > > > > > > > > this > > > > > > > > >> > is a > > > > > > > > >> > >> >> > number > > > > > > > > >> > >> >> > > > > > > > independent > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > parameter. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (2) We manually allocate > > and > > > > > > > > de-allocate > > > > > > > > >> > the > > > > > > > > >> > >> >> memory > > > > > > > > >> > >> >> > > for > > > > > > > > >> > >> >> > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > MemorySegments > > > > > > > > >> > >> >> > > > > > > > > > > > > > (option 2). That way we > > need > > > > not > > > > > > > worry > > > > > > > > >> > about > > > > > > > > >> > >> >> > > triggering > > > > > > > > >> > >> >> > > > > GC > > > > > > > > >> > >> >> > > > > > by > > > > > > > > >> > >> >> > > > > > > > > some > > > > > > > > >> > >> >> > > > > > > > > > > > > > threshold or > bookkeeping, > > > but > > > > it > > > > > > is > > > > > > > > >> harder > > > > > > > > >> > to > > > > > > > > >> > >> >> > prevent > > > > > > > > >> > >> >> > > > > > > > segfaults. > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > be very careful about > when > > > we > > > > > > > release > > > > > > > > >> the > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > > > segments > > > > > > > > >> > >> >> > > > > > > (only > > > > > > > > >> > >> >> > > > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > cleanup phase of the > main > > > > > thread). > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option > 1.1, > > we > > > > > > > probably > > > > > > > > >> need > > > > > > > > >> > to > > > > > > > > >> > >> >> set > > > > > > > > >> > >> >> > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > to > > > > > > > > >> > >> >> > > "off_heap_managed_memory + > > > > > > > > >> > >> >> > > > > > > > > > > direct_memory" > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > have "direct_memory" as > a > > > > > separate > > > > > > > > >> reserved > > > > > > > > >> > >> >> memory > > > > > > > > >> > >> >> > > > pool. > > > > > > > > >> > >> >> > > > > > > > Because > > > > > > > > >> > >> >> > > > > > > > > if > > > > > > > > >> > >> >> > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > just > > > > > > > > >> > >> >> > > > > > > > > > > > > > set > > > "-XX:MaxDirectMemorySize" > > > > to > > > > > > > > >> > >> >> > > > > "off_heap_managed_memory + > > > > > > > > >> > >> >> > > > > > > > > > > > > jvm_overhead", > > > > > > > > >> > >> >> > > > > > > > > > > > > > then there will be times > > > when > > > > > that > > > > > > > > >> entire > > > > > > > > >> > >> >> memory is > > > > > > > > >> > >> >> > > > > > allocated > > > > > > > > >> > >> >> > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > buffers and we have > > nothing > > > > left > > > > > > for > > > > > > > > the > > > > > > > > >> > JVM > > > > > > > > >> > >> >> > > overhead. > > > > > > > > >> > >> >> > > > So > > > > > > > > >> > >> >> > > > > > we > > > > > > > > >> > >> >> > > > > > > > > either > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > >> > >> >> > > > > > > > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > > > > > > way to compensate for > that > > > > > (again > > > > > > > some > > > > > > > > >> > safety > > > > > > > > >> > >> >> > margin > > > > > > > > >> > >> >> > > > > cutoff > > > > > > > > >> > >> >> > > > > > > > > value) > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > will exceed container > > > memory. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option > 1.2, > > we > > > > > need > > > > > > to > > > > > > > > be > > > > > > > > >> > aware > > > > > > > > >> > >> >> that > > > > > > > > >> > >> >> > it > > > > > > > > >> > >> >> > > > > takes > > > > > > > > >> > >> >> > > > > > > > > > elaborate > > > > > > > > >> > >> >> > > > > > > > > > > > > logic > > > > > > > > >> > >> >> > > > > > > > > > > > > > to push recycling of > > direct > > > > > > buffers > > > > > > > > >> without > > > > > > > > >> > >> >> always > > > > > > > > >> > >> >> > > > > > > triggering a > > > > > > > > >> > >> >> > > > > > > > > > full > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My first guess is that > the > > > > > options > > > > > > > > will > > > > > > > > >> be > > > > > > > > >> > >> >> easiest > > > > > > > > >> > >> >> > to > > > > > > > > >> > >> >> > > > do > > > > > > > > >> > >> >> > > > > in > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > following > > > > > > > > >> > >> >> > > > > > > > > > > > > > order: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.1 with a > > > > dedicated > > > > > > > > >> > direct_memory > > > > > > > > >> > >> >> > > > parameter, > > > > > > > > >> > >> >> > > > > as > > > > > > > > >> > >> >> > > > > > > > > > discussed > > > > > > > > >> > >> >> > > > > > > > > > > > > > above. We would need to > > > find a > > > > > way > > > > > > > to > > > > > > > > >> set > > > > > > > > >> > the > > > > > > > > >> > >> >> > > > > direct_memory > > > > > > > > >> > >> >> > > > > > > > > > parameter > > > > > > > > >> > >> >> > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > default. We could start > > with > > > > 64 > > > > > MB > > > > > > > and > > > > > > > > >> see > > > > > > > > >> > >> how > > > > > > > > >> > >> >> it > > > > > > > > >> > >> >> > > goes > > > > > > > > >> > >> >> > > > in > > > > > > > > >> > >> >> > > > > > > > > practice. > > > > > > > > >> > >> >> > > > > > > > > > > One > > > > > > > > >> > >> >> > > > > > > > > > > > > > danger I see is that > > setting > > > > > this > > > > > > > loo > > > > > > > > >> low > > > > > > > > >> > can > > > > > > > > >> > >> >> > cause a > > > > > > > > >> > >> >> > > > > bunch > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > additional > > > > > > > > >> > >> >> > > > > > > > > > > > > > GCs compared to before > (we > > > > need > > > > > to > > > > > > > > watch > > > > > > > > >> > this > > > > > > > > >> > >> >> > > > carefully). > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 2. It is > > actually > > > > > quite > > > > > > > > >> simple > > > > > > > > >> > to > > > > > > > > >> > >> >> > > implement, > > > > > > > > >> > >> >> > > > > we > > > > > > > > >> > >> >> > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > try > > > > > > > > >> > >> >> > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > segfault safe we are at > > the > > > > > > moment. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.2: We would > > not > > > > > touch > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > >> > >> >> > > > > > > > > > > > > parameter > > > > > > > > >> > >> >> > > > > > > > > > > > > > at all and assume that > all > > > the > > > > > > > direct > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > > allocations > > > > > > > > >> > >> >> > > > > > that > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > Netty do are infrequent > > > enough > > > > > to > > > > > > be > > > > > > > > >> > cleaned > > > > > > > > >> > >> up > > > > > > > > >> > >> >> > fast > > > > > > > > >> > >> >> > > > > enough > > > > > > > > >> > >> >> > > > > > > > > through > > > > > > > > >> > >> >> > > > > > > > > > > > > regular > > > > > > > > >> > >> >> > > > > > > > > > > > > > GC. I am not sure if > that > > > is a > > > > > > valid > > > > > > > > >> > >> assumption, > > > > > > > > >> > >> >> > > > though. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Best, > > > > > > > > >> > >> >> > > > > > > > > > > > > > Stephan > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at > > 2:16 > > > > PM > > > > > > > > Xintong > > > > > > > > >> > Song > > > > > > > > >> > >> < > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing > your > > > > > opinion > > > > > > > > Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor of > > > > > alternative > > > > > > > 2. > > > > > > > > I > > > > > > > > >> was > > > > > > > > >> > >> >> > wondering > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > Unsafe.allocate() > > > for > > > > > > > off-heap > > > > > > > > >> > >> managed > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > > and > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But > after > > > > > giving > > > > > > > it a > > > > > > > > >> > second > > > > > > > > >> > >> >> > > thought, > > > > > > > > >> > >> >> > > > I > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 using > > direct > > > > > > memory > > > > > > > > for > > > > > > > > >> > >> off-heap > > > > > > > > >> > >> >> > > > managed > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > concern, > > I > > > > > think > > > > > > > what > > > > > > > > >> > >> proposed > > > > > > > > >> > >> >> in > > > > > > > > >> > >> >> > > this > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > memory > > > and > > > > > > > network > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > allocated > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), > which > > > > means > > > > > > > they > > > > > > > > >> are > > > > > > > > >> > >> >> > practically > > > > > > > > >> > >> >> > > > > > native > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > > direct > > > > > > memory. > > > > > > > > The > > > > > > > > >> > only > > > > > > > > >> > >> >> parts > > > > > > > > >> > >> >> > of > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory are > > task > > > > > > > off-heap > > > > > > > > >> > memory > > > > > > > > >> > >> and > > > > > > > > >> > >> >> > JVM > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly alternative 2 > > > > suggests > > > > > > to > > > > > > > > set > > > > > > > > >> the > > > > > > > > >> > >> JVM > > > > > > > > >> > >> >> max > > > > > > > > >> > >> >> > > > > direct > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > at > > > 1:48 > > > > > PM > > > > > > > Till > > > > > > > > >> > >> Rohrmann > > > > > > > > >> > >> >> < > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > clarification > > > > > > > > >> Xintong. I > > > > > > > > >> > >> >> > > understand > > > > > > > > >> > >> >> > > > > the > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in favour > > of > > > > > > option 2 > > > > > > > > >> > because > > > > > > > > >> > >> it > > > > > > > > >> > >> >> > makes > > > > > > > > >> > >> >> > > > > > things > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the > direct > > > > > > memory, I > > > > > > > > >> fear > > > > > > > > >> > >> that > > > > > > > > >> > >> >> we > > > > > > > > >> > >> >> > > might > > > > > > > > >> > >> >> > > > > end > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we are > > > > > currently > > > > > > > in: > > > > > > > > >> The > > > > > > > > >> > >> user > > > > > > > > >> > >> >> > might > > > > > > > > >> > >> >> > > > see > > > > > > > > >> > >> >> > > > > > that > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS and > > > does > > > > > not > > > > > > > know > > > > > > > > >> why > > > > > > > > >> > >> this > > > > > > > > >> > >> >> is > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > the > > > > > process > > > > > > > > memory > > > > > > > > >> > size > > > > > > > > >> > >> >> > > (similar > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order to > > > > > accommodate > > > > > > > for > > > > > > > > >> the > > > > > > > > >> > >> extra > > > > > > > > >> > >> >> > > direct > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > > memory > > > > > > budgets > > > > > > > > >> which > > > > > > > > >> > >> are > > > > > > > > >> > >> >> not > > > > > > > > >> > >> >> > > > fully > > > > > > > > >> > >> >> > > > > > used > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the overall > > > memory > > > > > > > > >> consumption. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > > 11:01 > > > > > > AM > > > > > > > > >> > Xintong > > > > > > > > >> > >> >> Song < > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain > this > > > > with a > > > > > > > > >> concrete > > > > > > > > >> > >> >> example > > > > > > > > >> > >> >> > > Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we have > > the > > > > > > > following > > > > > > > > >> > >> scenario. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > > Memory: > > > > 1GB > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct Memory > > > (Task > > > > > > > Off-Heap > > > > > > > > >> > >> Memory + > > > > > > > > >> > >> >> JVM > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory (JVM > > Heap > > > > > > Memory, > > > > > > > > JVM > > > > > > > > >> > >> >> Metaspace, > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network Memory): > > 800MB > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 2, > > we > > > > set > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 3, > > we > > > > set > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > a > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > direct > > > > > memory > > > > > > > > usage > > > > > > > > >> of > > > > > > > > >> > >> Task > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed > 200MB, > > > > then > > > > > > > > >> > alternative 2 > > > > > > > > >> > >> >> and > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > > Setting > > > > > larger > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the other > > > > memory > > > > > > > pools. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > direct > > > > > memory > > > > > > > > usage > > > > > > > > >> of > > > > > > > > >> > >> Task > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > potentially > > > > > exceed > > > > > > > > 200MB, > > > > > > > > >> > then > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - Alternative 2 > > > > suffers > > > > > > > from > > > > > > > > >> > >> frequent > > > > > > > > >> > >> >> OOM. > > > > > > > > >> > >> >> > > To > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do is > to > > > > > modify > > > > > > > the > > > > > > > > >> > >> >> configuration > > > > > > > > >> > >> >> > > and > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task Off-Heap > > > > Memory + > > > > > > JVM > > > > > > > > >> > >> Overhead). > > > > > > > > >> > >> >> > Let's > > > > > > > > >> > >> >> > > > say > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct Memory > to > > > > 250MB, > > > > > > > this > > > > > > > > >> will > > > > > > > > >> > >> >> reduce > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > > total > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to 750MB, > > > given > > > > > the > > > > > > > > total > > > > > > > > >> > >> process > > > > > > > > >> > >> >> > > memory > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > alternative > > > 3, > > > > > > there > > > > > > > is > > > > > > > > >> no > > > > > > > > >> > >> >> chance of > > > > > > > > >> > >> >> > > > > direct > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of exceeding > the > > > > total > > > > > > > > process > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > limit, > > > > > > > > >> > >> >> > > > but > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up all > > the > > > > > > reserved > > > > > > > > >> native > > > > > > > > >> > >> >> memory > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > > Metaspace), > > > > > > if > > > > > > > > the > > > > > > > > >> > >> actual > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very close > to > > > > > 200MB, > > > > > > > user > > > > > > > > >> > >> probably > > > > > > > > >> > >> >> do > > > > > > > > >> > >> >> > > not > > > > > > > > >> > >> >> > > > > need > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > configurations. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I think > > > from > > > > > the > > > > > > > > user's > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for alternative 2 > > may > > > > lead > > > > > > to > > > > > > > > >> lower > > > > > > > > >> > >> >> resource > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > > 10:28 > > > > > > > AM > > > > > > > > >> Till > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > >> > >> >> > > < > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you have > > to > > > > help > > > > > > me > > > > > > > > >> > >> understand > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > > memory > > > > > under > > > > > > > > >> > utilization > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: > > set > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > Task > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. Then > > there > > > > is > > > > > > the > > > > > > > > risk > > > > > > > > >> > that > > > > > > > > >> > >> >> this > > > > > > > > >> > >> >> > > size > > > > > > > > >> > >> >> > > > > is > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of garbage > > > > > collection > > > > > > > and > > > > > > > > >> > >> >> potentially > > > > > > > > >> > >> >> > an > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 3: > > set > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 2. > > This > > > > > would > > > > > > of > > > > > > > > >> course > > > > > > > > >> > >> >> reduce > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > alternative > > > > 2 > > > > > > now > > > > > > > > >> result > > > > > > > > >> > >> in an > > > > > > > > >> > >> >> > > under > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > > alternative > > > > > 3? > > > > > > > If > > > > > > > > >> > >> >> alternative 3 > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct memory > size > > > and > > > > > we > > > > > > > use > > > > > > > > >> only > > > > > > > > >> > >> >> little, > > > > > > > > >> > >> >> > > > then I > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 3 > > > results > > > > in > > > > > > > > memory > > > > > > > > >> > under > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, > > 2019 > > > > at > > > > > > 4:19 > > > > > > > > PM > > > > > > > > >> > Yang > > > > > > > > >> > >> >> Wang < > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > xintong,till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native and > > > Direct > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point is > > > setting > > > > a > > > > > > very > > > > > > > > >> large > > > > > > > > >> > >> max > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > differentiate > > > direct > > > > > and > > > > > > > > >> native > > > > > > > > >> > >> >> memory. > > > > > > > > >> > >> >> > If > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct memory > > and > > > > > > > framework > > > > > > > > >> > direct > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > >> > >> >> > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in favor > of > > > > > setting > > > > > > > > >> direct > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > with > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > Calculation > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree with > > > > xintong. > > > > > > For > > > > > > > > Yarn > > > > > > > > >> > and > > > > > > > > >> > >> >> k8s,we > > > > > > > > >> > >> >> > > > need > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configurations > > in > > > > > client > > > > > > > to > > > > > > > > >> avoid > > > > > > > > >> > >> >> > > submitting > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > > master. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < > > > > > > > > >> > >> [hidden email] > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks for > > > > replying, > > > > > > > Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > MemorySegment, I > > > > > > > > think > > > > > > > > >> > you > > > > > > > > >> > >> are > > > > > > > > >> > >> >> > > right > > > > > > > > >> > >> >> > > > > that > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in the > > > scope > > > > > of > > > > > > > this > > > > > > > > >> > FLIP. > > > > > > > > >> > >> >> This > > > > > > > > >> > >> >> > > FLIP > > > > > > > > >> > >> >> > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configure > > memory > > > > > pools > > > > > > > for > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > >> > >> >> > > > > with > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > consumers > > > > use > > > > > > it. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About direct > > > > > memory, I > > > > > > > > think > > > > > > > > >> > >> >> > alternative > > > > > > > > >> > >> >> > > 3 > > > > > > > > >> > >> >> > > > > may > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > reservation > > > issue > > > > > that > > > > > > > > >> > >> alternative 2 > > > > > > > > >> > >> >> > > does, > > > > > > > > >> > >> >> > > > > but > > > > > > > > >> > >> >> > > > > > at > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using memory > > at > > > > the > > > > > > > > >> container > > > > > > > > >> > >> level, > > > > > > > > >> > >> >> > > which > > > > > > > > >> > >> >> > > > is > > > > > > > > >> > >> >> > > > > > not > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both "Task > > > > Off-Heap > > > > > > > > Memory" > > > > > > > > >> and > > > > > > > > >> > >> "JVM > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative > 2, > > > > users > > > > > > > might > > > > > > > > >> > >> configure > > > > > > > > >> > >> >> > them > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to > avoid > > > > > getting > > > > > > a > > > > > > > > >> direct > > > > > > > > >> > >> OOM. > > > > > > > > >> > >> >> For > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct OOM, > so > > > > they > > > > > > may > > > > > > > > not > > > > > > > > >> > >> config > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > two > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > consequences > > > > are > > > > > > > risks > > > > > > > > >> of > > > > > > > > >> > >> >> overall > > > > > > > > >> > >> >> > > > > container > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug > > 13, > > > > 2019 > > > > > > at > > > > > > > > >> 9:39 AM > > > > > > > > >> > >> Till > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for > > > > > proposing > > > > > > > > this > > > > > > > > >> > FLIP > > > > > > > > >> > >> >> > Xintong. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in > all I > > > > think > > > > > > it > > > > > > > > >> already > > > > > > > > >> > >> >> looks > > > > > > > > >> > >> >> > > quite > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > question > > about > > > > > > > > allocating > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > > > segments, > > > > > > > > >> > >> >> > > > > I > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > strictly > > > > necessary > > > > > > to > > > > > > > do > > > > > > > > >> in > > > > > > > > >> > the > > > > > > > > >> > >> >> > context > > > > > > > > >> > >> >> > > > of > > > > > > > > >> > >> >> > > > > > this > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be done > as a > > > > > follow > > > > > > > up? > > > > > > > > >> > Without > > > > > > > > >> > >> >> > knowing > > > > > > > > >> > >> >> > > > all > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that we > > would > > > > > widen > > > > > > > the > > > > > > > > >> scope > > > > > > > > >> > >> of > > > > > > > > >> > >> >> this > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > >> > >> >> > > > > > too > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to touch > all > > > the > > > > > > > > existing > > > > > > > > >> > call > > > > > > > > >> > >> >> sites > > > > > > > > >> > >> >> > of > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > segments > > > > > > (this > > > > > > > > >> should > > > > > > > > >> > >> >> mainly > > > > > > > > >> > >> >> > be > > > > > > > > >> > >> >> > > > > batch > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the memory > > > > > > reservation > > > > > > > > >> call > > > > > > > > >> > to > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this and I > > > would > > > > > > hope > > > > > > > > that > > > > > > > > >> > >> this is > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > only > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > streaming > > job > > > > > would > > > > > > > have > > > > > > > > >> with > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Concerning > > the > > > > > > second > > > > > > > > open > > > > > > > > >> > >> >> question > > > > > > > > >> > >> >> > > about > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > memory > > > > > > limit, I > > > > > > > > >> would > > > > > > > > >> > >> also > > > > > > > > >> > >> >> be > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > leaving it > > > open > > > > > > would > > > > > > > be > > > > > > > > >> > best. > > > > > > > > >> > >> My > > > > > > > > >> > >> >> > > concern > > > > > > > > >> > >> >> > > > > > about > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in a > > > similar > > > > > > > > situation > > > > > > > > >> as > > > > > > > > >> > we > > > > > > > > >> > >> >> are > > > > > > > > >> > >> >> > now > > > > > > > > >> > >> >> > > > > with > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > different > > > > > memory > > > > > > > > pools > > > > > > > > >> > are > > > > > > > > >> > >> not > > > > > > > > >> > >> >> > > > clearly > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > different > > > > pool, > > > > > > then > > > > > > > > it > > > > > > > > >> is > > > > > > > > >> > >> quite > > > > > > > > >> > >> >> > hard > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > process to > > get > > > > > > killed > > > > > > > > for > > > > > > > > >> > using > > > > > > > > >> > >> >> too > > > > > > > > >> > >> >> > > much > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead to a > > > > similar > > > > > > > > >> situation > > > > > > > > >> > >> what > > > > > > > > >> > >> >> we > > > > > > > > >> > >> >> > > have > > > > > > > > >> > >> >> > > > > with > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > setting a > > sane > > > > > > default > > > > > > > > >> value > > > > > > > > >> > >> for > > > > > > > > >> > >> >> max > > > > > > > > >> > >> >> > > > direct > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option to > > > > increase > > > > > > it > > > > > > > if > > > > > > > > >> he > > > > > > > > >> > >> runs > > > > > > > > >> > >> >> into > > > > > > > > >> > >> >> > > an > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, > > how > > > > > would > > > > > > > > >> > >> alternative 2 > > > > > > > > >> > >> >> > lead > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > alternative > > 3 > > > > > where > > > > > > we > > > > > > > > set > > > > > > > > >> > the > > > > > > > > >> > >> >> direct > > > > > > > > >> > >> >> > > > > memory > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, > Aug > > 9, > > > > > 2019 > > > > > > at > > > > > > > > >> 9:12 > > > > > > > > >> > AM > > > > > > > > >> > >> >> > Xintong > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks > for > > > the > > > > > > > > feedback, > > > > > > > > >> > >> Yang. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Regarding > > > your > > > > > > > > comments: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Native > > and > > > > > Direct > > > > > > > > >> Memory* > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > > > setting > > > > > a > > > > > > > very > > > > > > > > >> > large > > > > > > > > >> > >> max > > > > > > > > >> > >> >> > > direct > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > > sides. > > > > > E.g., > > > > > > we > > > > > > > > do > > > > > > > > >> not > > > > > > > > >> > >> >> worry > > > > > > > > >> > >> >> > > about > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > allocate > > > > > > managed > > > > > > > / > > > > > > > > >> > network > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > However, > > > there > > > > > are > > > > > > > > also > > > > > > > > >> > some > > > > > > > > >> > >> >> down > > > > > > > > >> > >> >> > > sides > > > > > > > > >> > >> >> > > > > of > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - One > > > > thing I > > > > > > can > > > > > > > > >> think > > > > > > > > >> > >> of is > > > > > > > > >> > >> >> > that > > > > > > > > >> > >> >> > > > if > > > > > > > > >> > >> >> > > > > a > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > killed > > > due > > > > to > > > > > > > > >> overusing > > > > > > > > >> > >> >> memory, > > > > > > > > >> > >> >> > it > > > > > > > > >> > >> >> > > > > could > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > of > the > > > > memory > > > > > > is > > > > > > > > >> > overused. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > Another > > > > > down > > > > > > > side > > > > > > > > >> is > > > > > > > > >> > >> that > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > JVM > > > > > > > > >> > >> >> > > > > > never > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > direct > > > > memory > > > > > > > > limit, > > > > > > > > >> > >> because > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > limit > > > > > > > > >> > >> >> > > > > > is > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > means > > we > > > > kind > > > > > > of > > > > > > > > >> relay > > > > > > > > >> > on > > > > > > > > >> > >> >> heap > > > > > > > > >> > >> >> > > > memory > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > memory. > > > > That > > > > > > > could > > > > > > > > >> be a > > > > > > > > >> > >> >> problem > > > > > > > > >> > >> >> > in > > > > > > > > >> > >> >> > > > > cases > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > usage > > but > > > > not > > > > > > > > enough > > > > > > > > >> > heap > > > > > > > > >> > >> >> > activity > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe > you > > > can > > > > > > share > > > > > > > > your > > > > > > > > >> > >> reasons > > > > > > > > >> > >> >> > for > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if there > > are > > > > > > > anything > > > > > > > > >> else > > > > > > > > >> > I > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > Calculation* > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If there > > is > > > > any > > > > > > > > conflict > > > > > > > > >> > >> between > > > > > > > > >> > >> >> > > > multiple > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > specified, > > > > > > > > I > > > > > > > > >> > >> think we > > > > > > > > >> > >> >> > > should > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > > doing > > > > > > > checking > > > > > > > > >> on > > > > > > > > >> > the > > > > > > > > >> > >> >> > client > > > > > > > > >> > >> >> > > > side > > > > > > > > >> > >> >> > > > > > is > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we > can > > > > > > discover > > > > > > > > the > > > > > > > > >> > >> problem > > > > > > > > >> > >> >> > > before > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > always > > a > > > > good > > > > > > > > thing. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But we > can > > > not > > > > > > only > > > > > > > > >> rely on > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > client > > > > > > > > >> > >> >> > > > > side > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > cluster > > > > > > > > >> > >> TaskManagers > > > > > > > > >> > >> >> on > > > > > > > > >> > >> >> > > > > > different > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > different > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > and > > > > > > > the > > > > > > > > >> > client > > > > > > > > >> > >> >> does > > > > > > > > >> > >> >> > > see > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What do > > you > > > > > think? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thank > you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong > > Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, > > Aug > > > 8, > > > > > > 2019 > > > > > > > at > > > > > > > > >> 5:09 > > > > > > > > >> > >> PM > > > > > > > > >> > >> >> Yang > > > > > > > > >> > >> >> > > > Wang > > > > > > > > >> > >> >> > > > > < > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > > xintong, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > > your > > > > > > > > >> detailed > > > > > > > > >> > >> >> > proposal. > > > > > > > > >> > >> >> > > > > After > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > it > > > > > > > will > > > > > > > > be > > > > > > > > >> > more > > > > > > > > >> > >> >> > > powerful > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > just > > have > > > > few > > > > > > > > >> questions > > > > > > > > >> > >> about > > > > > > > > >> > >> >> it. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > Native > > > > > and > > > > > > > > Direct > > > > > > > > >> > >> Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We do > > not > > > > > > > > >> differentiate > > > > > > > > >> > >> user > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > included > > > in > > > > > task > > > > > > > > >> off-heap > > > > > > > > >> > >> >> memory. > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > Memory > > > > > > > > >> Calculation > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If the > > sum > > > > of > > > > > > and > > > > > > > > >> > >> fine-grained > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > larger > > > > than > > > > > > > total > > > > > > > > >> > >> process > > > > > > > > >> > >> >> > > memory, > > > > > > > > >> > >> >> > > > > how > > > > > > > > >> > >> >> > > > > > do > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we > need > > to > > > > > check > > > > > > > the > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song < > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > everyone, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > > would > > > > > like > > > > > > to > > > > > > > > >> start > > > > > > > > >> > a > > > > > > > > >> > >> >> > > discussion > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > for > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > memory > > > > > > > > >> > >> >> > > configurations. > > > > > > > > >> > >> >> > > > > The > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > early > > > > design > > > > > > > > "Memory > > > > > > > > >> > >> >> Management > > > > > > > > >> > >> >> > > and > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > with > > > > updates > > > > > > > from > > > > > > > > >> > >> follow-up > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > This > > > FLIP > > > > > > > > addresses > > > > > > > > >> > >> several > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > memory > > > > > > > > >> > >> >> > > configuration. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Different > > > > > > > > >> > >> configuration > > > > > > > > >> > >> >> > for > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Complex > > > > > > and > > > > > > > > >> > >> difficult > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Complicated, > > > > > > > > >> > >> uncertain > > > > > > > > >> > >> >> and > > > > > > > > >> > >> >> > > > hard > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key > > > > changes > > > > > to > > > > > > > > solve > > > > > > > > >> > the > > > > > > > > >> > >> >> > problems > > > > > > > > >> > >> >> > > > can > > > > > > > > >> > >> >> > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Extend > > > > > > > memory > > > > > > > > >> > >> manager > > > > > > > > >> > >> >> to > > > > > > > > >> > >> >> > > also > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Modify > > > > > > how > > > > > > > > >> > >> TaskExecutor > > > > > > > > >> > >> >> > > memory > > > > > > > > >> > >> >> > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > >> reservations > > > > > > > > >> > >> and > > > > > > > > >> > >> >> > pools. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Simplify > > > > > > > > memory > > > > > > > > >> > >> >> > > configuration > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > find > > > > > > more > > > > > > > > >> > details > > > > > > > > >> > >> in > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > note > > > > > > > that > > > > > > > > >> the > > > > > > > > >> > >> early > > > > > > > > >> > >> >> > > design > > > > > > > > >> > >> >> > > > > doc > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > to > > > > > > > > have > > > > > > > > >> the > > > > > > > > >> > >> >> > > discussion > > > > > > > > >> > >> >> > > > in > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > forward > > > > > > > to > > > > > > > > >> your > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 at > > 2:16 > > > > PM > > > > > > > > Xintong > > > > > > > > >> > Song > > > > > > > > >> > >> < > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing > your > > > > > opinion > > > > > > > > Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor of > > > > > alternative > > > > > > > 2. > > > > > > > > I > > > > > > > > >> was > > > > > > > > >> > >> >> > wondering > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > Unsafe.allocate() > > > for > > > > > > > off-heap > > > > > > > > >> > >> managed > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > > and > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But > after > > > > > giving > > > > > > > it a > > > > > > > > >> > second > > > > > > > > >> > >> >> > > thought, > > > > > > > > >> > >> >> > > > I > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 using > > direct > > > > > > memory > > > > > > > > for > > > > > > > > >> > >> off-heap > > > > > > > > >> > >> >> > > > managed > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > concern, > > I > > > > > think > > > > > > > what > > > > > > > > >> > >> proposed > > > > > > > > >> > >> >> in > > > > > > > > >> > >> >> > > this > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > memory > > > and > > > > > > > network > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > allocated > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), > which > > > > means > > > > > > > they > > > > > > > > >> are > > > > > > > > >> > >> >> > practically > > > > > > > > >> > >> >> > > > > > native > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > > direct > > > > > > memory. > > > > > > > > The > > > > > > > > >> > only > > > > > > > > >> > >> >> parts > > > > > > > > >> > >> >> > of > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory are > > task > > > > > > > off-heap > > > > > > > > >> > memory > > > > > > > > >> > >> and > > > > > > > > >> > >> >> > JVM > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly alternative 2 > > > > suggests > > > > > > to > > > > > > > > set > > > > > > > > >> the > > > > > > > > >> > >> JVM > > > > > > > > >> > >> >> max > > > > > > > > >> > >> >> > > > > direct > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > at > > > 1:48 > > > > > PM > > > > > > > Till > > > > > > > > >> > >> Rohrmann > > > > > > > > >> > >> >> < > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > clarification > > > > > > > > >> Xintong. I > > > > > > > > >> > >> >> > > understand > > > > > > > > >> > >> >> > > > > the > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in favour > > of > > > > > > option 2 > > > > > > > > >> > because > > > > > > > > >> > >> it > > > > > > > > >> > >> >> > makes > > > > > > > > >> > >> >> > > > > > things > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the > direct > > > > > > memory, I > > > > > > > > >> fear > > > > > > > > >> > >> that > > > > > > > > >> > >> >> we > > > > > > > > >> > >> >> > > might > > > > > > > > >> > >> >> > > > > end > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we are > > > > > currently > > > > > > > in: > > > > > > > > >> The > > > > > > > > >> > >> user > > > > > > > > >> > >> >> > might > > > > > > > > >> > >> >> > > > see > > > > > > > > >> > >> >> > > > > > that > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS and > > > does > > > > > not > > > > > > > know > > > > > > > > >> why > > > > > > > > >> > >> this > > > > > > > > >> > >> >> is > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > the > > > > > process > > > > > > > > memory > > > > > > > > >> > size > > > > > > > > >> > >> >> > > (similar > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order to > > > > > accommodate > > > > > > > for > > > > > > > > >> the > > > > > > > > >> > >> extra > > > > > > > > >> > >> >> > > direct > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > > memory > > > > > > budgets > > > > > > > > >> which > > > > > > > > >> > >> are > > > > > > > > >> > >> >> not > > > > > > > > >> > >> >> > > > fully > > > > > > > > >> > >> >> > > > > > used > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the overall > > > memory > > > > > > > > >> consumption. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > > 11:01 > > > > > > AM > > > > > > > > >> > Xintong > > > > > > > > >> > >> >> Song < > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain > this > > > > with a > > > > > > > > >> concrete > > > > > > > > >> > >> >> example > > > > > > > > >> > >> >> > > Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we have > > the > > > > > > > following > > > > > > > > >> > >> scenario. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > > Memory: > > > > 1GB > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct Memory > > > (Task > > > > > > > Off-Heap > > > > > > > > >> > >> Memory + > > > > > > > > >> > >> >> JVM > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory (JVM > > Heap > > > > > > Memory, > > > > > > > > JVM > > > > > > > > >> > >> >> Metaspace, > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network Memory): > > 800MB > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 2, > > we > > > > set > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative 3, > > we > > > > set > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > a > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > direct > > > > > memory > > > > > > > > usage > > > > > > > > >> of > > > > > > > > >> > >> Task > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed > 200MB, > > > > then > > > > > > > > >> > alternative 2 > > > > > > > > >> > >> >> and > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > > Setting > > > > > larger > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the other > > > > memory > > > > > > > pools. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > direct > > > > > memory > > > > > > > > usage > > > > > > > > >> of > > > > > > > > >> > >> Task > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > potentially > > > > > exceed > > > > > > > > 200MB, > > > > > > > > >> > then > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - Alternative 2 > > > > suffers > > > > > > > from > > > > > > > > >> > >> frequent > > > > > > > > >> > >> >> OOM. > > > > > > > > >> > >> >> > > To > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do is > to > > > > > modify > > > > > > > the > > > > > > > > >> > >> >> configuration > > > > > > > > >> > >> >> > > and > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task Off-Heap > > > > Memory + > > > > > > JVM > > > > > > > > >> > >> Overhead). > > > > > > > > >> > >> >> > Let's > > > > > > > > >> > >> >> > > > say > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct Memory > to > > > > 250MB, > > > > > > > this > > > > > > > > >> will > > > > > > > > >> > >> >> reduce > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > > total > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to 750MB, > > > given > > > > > the > > > > > > > > total > > > > > > > > >> > >> process > > > > > > > > >> > >> >> > > memory > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > alternative > > > 3, > > > > > > there > > > > > > > is > > > > > > > > >> no > > > > > > > > >> > >> >> chance of > > > > > > > > >> > >> >> > > > > direct > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of exceeding > the > > > > total > > > > > > > > process > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > limit, > > > > > > > > >> > >> >> > > > but > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up all > > the > > > > > > reserved > > > > > > > > >> native > > > > > > > > >> > >> >> memory > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > > Metaspace), > > > > > > if > > > > > > > > the > > > > > > > > >> > >> actual > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very close > to > > > > > 200MB, > > > > > > > user > > > > > > > > >> > >> probably > > > > > > > > >> > >> >> do > > > > > > > > >> > >> >> > > not > > > > > > > > >> > >> >> > > > > need > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > configurations. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I think > > > from > > > > > the > > > > > > > > user's > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for alternative 2 > > may > > > > lead > > > > > > to > > > > > > > > >> lower > > > > > > > > >> > >> >> resource > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > > 10:28 > > > > > > > AM > > > > > > > > >> Till > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > >> > >> >> > > < > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you have > > to > > > > help > > > > > > me > > > > > > > > >> > >> understand > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > > memory > > > > > under > > > > > > > > >> > utilization > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 2: > > set > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > Task > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. Then > > there > > > > is > > > > > > the > > > > > > > > risk > > > > > > > > >> > that > > > > > > > > >> > >> >> this > > > > > > > > >> > >> >> > > size > > > > > > > > >> > >> >> > > > > is > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of garbage > > > > > collection > > > > > > > and > > > > > > > > >> > >> >> potentially > > > > > > > > >> > >> >> > an > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative 3: > > set > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 2. > > This > > > > > would > > > > > > of > > > > > > > > >> course > > > > > > > > >> > >> >> reduce > > > > > > > > >> > >> >> > > the > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > alternative > > > > 2 > > > > > > now > > > > > > > > >> result > > > > > > > > >> > >> in an > > > > > > > > >> > >> >> > > under > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > > alternative > > > > > 3? > > > > > > > If > > > > > > > > >> > >> >> alternative 3 > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct memory > size > > > and > > > > > we > > > > > > > use > > > > > > > > >> only > > > > > > > > >> > >> >> little, > > > > > > > > >> > >> >> > > > then I > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 3 > > > results > > > > in > > > > > > > > memory > > > > > > > > >> > under > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug 13, > > 2019 > > > > at > > > > > > 4:19 > > > > > > > > PM > > > > > > > > >> > Yang > > > > > > > > >> > >> >> Wang < > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > xintong,till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native and > > > Direct > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point is > > > setting > > > > a > > > > > > very > > > > > > > > >> large > > > > > > > > >> > >> max > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > differentiate > > > direct > > > > > and > > > > > > > > >> native > > > > > > > > >> > >> >> memory. > > > > > > > > >> > >> >> > If > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct memory > > and > > > > > > > framework > > > > > > > > >> > direct > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > >> > >> >> > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in favor > of > > > > > setting > > > > > > > > >> direct > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > with > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > Calculation > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree with > > > > xintong. > > > > > > For > > > > > > > > Yarn > > > > > > > > >> > and > > > > > > > > >> > >> >> k8s,we > > > > > > > > >> > >> >> > > > need > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configurations > > in > > > > > client > > > > > > > to > > > > > > > > >> avoid > > > > > > > > >> > >> >> > > submitting > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > > master. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong Song < > > > > > > > > >> > >> [hidden email] > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks for > > > > replying, > > > > > > > Till. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > MemorySegment, I > > > > > > > > think > > > > > > > > >> > you > > > > > > > > >> > >> are > > > > > > > > >> > >> >> > > right > > > > > > > > >> > >> >> > > > > that > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in the > > > scope > > > > > of > > > > > > > this > > > > > > > > >> > FLIP. > > > > > > > > >> > >> >> This > > > > > > > > >> > >> >> > > FLIP > > > > > > > > >> > >> >> > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configure > > memory > > > > > pools > > > > > > > for > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > >> > >> >> > > > > with > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > consumers > > > > use > > > > > > it. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About direct > > > > > memory, I > > > > > > > > think > > > > > > > > >> > >> >> > alternative > > > > > > > > >> > >> >> > > 3 > > > > > > > > >> > >> >> > > > > may > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > reservation > > > issue > > > > > that > > > > > > > > >> > >> alternative 2 > > > > > > > > >> > >> >> > > does, > > > > > > > > >> > >> >> > > > > but > > > > > > > > >> > >> >> > > > > > at > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using memory > > at > > > > the > > > > > > > > >> container > > > > > > > > >> > >> level, > > > > > > > > >> > >> >> > > which > > > > > > > > >> > >> >> > > > is > > > > > > > > >> > >> >> > > > > > not > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both "Task > > > > Off-Heap > > > > > > > > Memory" > > > > > > > > >> and > > > > > > > > >> > >> "JVM > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative > 2, > > > > users > > > > > > > might > > > > > > > > >> > >> configure > > > > > > > > >> > >> >> > them > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to > avoid > > > > > getting > > > > > > a > > > > > > > > >> direct > > > > > > > > >> > >> OOM. > > > > > > > > >> > >> >> For > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct OOM, > so > > > > they > > > > > > may > > > > > > > > not > > > > > > > > >> > >> config > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > two > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > consequences > > > > are > > > > > > > risks > > > > > > > > >> of > > > > > > > > >> > >> >> overall > > > > > > > > >> > >> >> > > > > container > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, Aug > > 13, > > > > 2019 > > > > > > at > > > > > > > > >> 9:39 AM > > > > > > > > >> > >> Till > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks for > > > > > proposing > > > > > > > > this > > > > > > > > >> > FLIP > > > > > > > > >> > >> >> > Xintong. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in > all I > > > > think > > > > > > it > > > > > > > > >> already > > > > > > > > >> > >> >> looks > > > > > > > > >> > >> >> > > quite > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > question > > about > > > > > > > > allocating > > > > > > > > >> > >> memory > > > > > > > > >> > >> >> > > > segments, > > > > > > > > >> > >> >> > > > > I > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > strictly > > > > necessary > > > > > > to > > > > > > > do > > > > > > > > >> in > > > > > > > > >> > the > > > > > > > > >> > >> >> > context > > > > > > > > >> > >> >> > > > of > > > > > > > > >> > >> >> > > > > > this > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be done > as a > > > > > follow > > > > > > > up? > > > > > > > > >> > Without > > > > > > > > >> > >> >> > knowing > > > > > > > > >> > >> >> > > > all > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that we > > would > > > > > widen > > > > > > > the > > > > > > > > >> scope > > > > > > > > >> > >> of > > > > > > > > >> > >> >> this > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > >> > >> >> > > > > > too > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to touch > all > > > the > > > > > > > > existing > > > > > > > > >> > call > > > > > > > > >> > >> >> sites > > > > > > > > >> > >> >> > of > > > > > > > > >> > >> >> > > > the > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > segments > > > > > > (this > > > > > > > > >> should > > > > > > > > >> > >> >> mainly > > > > > > > > >> > >> >> > be > > > > > > > > >> > >> >> > > > > batch > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the memory > > > > > > reservation > > > > > > > > >> call > > > > > > > > >> > to > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this and I > > > would > > > > > > hope > > > > > > > > that > > > > > > > > >> > >> this is > > > > > > > > >> > >> >> > the > > > > > > > > >> > >> >> > > > only > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > streaming > > job > > > > > would > > > > > > > have > > > > > > > > >> with > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Concerning > > the > > > > > > second > > > > > > > > open > > > > > > > > >> > >> >> question > > > > > > > > >> > >> >> > > about > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > memory > > > > > > limit, I > > > > > > > > >> would > > > > > > > > >> > >> also > > > > > > > > >> > >> >> be > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > leaving it > > > open > > > > > > would > > > > > > > be > > > > > > > > >> > best. > > > > > > > > >> > >> My > > > > > > > > >> > >> >> > > concern > > > > > > > > >> > >> >> > > > > > about > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in a > > > similar > > > > > > > > situation > > > > > > > > >> as > > > > > > > > >> > we > > > > > > > > >> > >> >> are > > > > > > > > >> > >> >> > now > > > > > > > > >> > >> >> > > > > with > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > different > > > > > memory > > > > > > > > pools > > > > > > > > >> > are > > > > > > > > >> > >> not > > > > > > > > >> > >> >> > > > clearly > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > different > > > > pool, > > > > > > then > > > > > > > > it > > > > > > > > >> is > > > > > > > > >> > >> quite > > > > > > > > >> > >> >> > hard > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > process to > > get > > > > > > killed > > > > > > > > for > > > > > > > > >> > using > > > > > > > > >> > >> >> too > > > > > > > > >> > >> >> > > much > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead to a > > > > similar > > > > > > > > >> situation > > > > > > > > >> > >> what > > > > > > > > >> > >> >> we > > > > > > > > >> > >> >> > > have > > > > > > > > >> > >> >> > > > > with > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > setting a > > sane > > > > > > default > > > > > > > > >> value > > > > > > > > >> > >> for > > > > > > > > >> > >> >> max > > > > > > > > >> > >> >> > > > direct > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option to > > > > increase > > > > > > it > > > > > > > if > > > > > > > > >> he > > > > > > > > >> > >> runs > > > > > > > > >> > >> >> into > > > > > > > > >> > >> >> > > an > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > @Xintong, > > how > > > > > would > > > > > > > > >> > >> alternative 2 > > > > > > > > >> > >> >> > lead > > > > > > > > >> > >> >> > > to > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > alternative > > 3 > > > > > where > > > > > > we > > > > > > > > set > > > > > > > > >> > the > > > > > > > > >> > >> >> direct > > > > > > > > >> > >> >> > > > > memory > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, > Aug > > 9, > > > > > 2019 > > > > > > at > > > > > > > > >> 9:12 > > > > > > > > >> > AM > > > > > > > > >> > >> >> > Xintong > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks > for > > > the > > > > > > > > feedback, > > > > > > > > >> > >> Yang. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Regarding > > > your > > > > > > > > comments: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Native > > and > > > > > Direct > > > > > > > > >> Memory* > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > > > setting > > > > > a > > > > > > > very > > > > > > > > >> > large > > > > > > > > >> > >> max > > > > > > > > >> > >> >> > > direct > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > > sides. > > > > > E.g., > > > > > > we > > > > > > > > do > > > > > > > > >> not > > > > > > > > >> > >> >> worry > > > > > > > > >> > >> >> > > about > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > allocate > > > > > > managed > > > > > > > / > > > > > > > > >> > network > > > > > > > > >> > >> >> > memory > > > > > > > > >> > >> >> > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > However, > > > there > > > > > are > > > > > > > > also > > > > > > > > >> > some > > > > > > > > >> > >> >> down > > > > > > > > >> > >> >> > > sides > > > > > > > > >> > >> >> > > > > of > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - One > > > > thing I > > > > > > can > > > > > > > > >> think > > > > > > > > >> > >> of is > > > > > > > > >> > >> >> > that > > > > > > > > >> > >> >> > > > if > > > > > > > > >> > >> >> > > > > a > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > killed > > > due > > > > to > > > > > > > > >> overusing > > > > > > > > >> > >> >> memory, > > > > > > > > >> > >> >> > it > > > > > > > > >> > >> >> > > > > could > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > of > the > > > > memory > > > > > > is > > > > > > > > >> > overused. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > Another > > > > > down > > > > > > > side > > > > > > > > >> is > > > > > > > > >> > >> that > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > JVM > > > > > > > > >> > >> >> > > > > > never > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > direct > > > > memory > > > > > > > > limit, > > > > > > > > >> > >> because > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > limit > > > > > > > > >> > >> >> > > > > > is > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > means > > we > > > > kind > > > > > > of > > > > > > > > >> relay > > > > > > > > >> > on > > > > > > > > >> > >> >> heap > > > > > > > > >> > >> >> > > > memory > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > memory. > > > > That > > > > > > > could > > > > > > > > >> be a > > > > > > > > >> > >> >> problem > > > > > > > > >> > >> >> > in > > > > > > > > >> > >> >> > > > > cases > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > usage > > but > > > > not > > > > > > > > enough > > > > > > > > >> > heap > > > > > > > > >> > >> >> > activity > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe > you > > > can > > > > > > share > > > > > > > > your > > > > > > > > >> > >> reasons > > > > > > > > >> > >> >> > for > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if there > > are > > > > > > > anything > > > > > > > > >> else > > > > > > > > >> > I > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > Calculation* > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If there > > is > > > > any > > > > > > > > conflict > > > > > > > > >> > >> between > > > > > > > > >> > >> >> > > > multiple > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > specified, > > > > > > > > I > > > > > > > > >> > >> think we > > > > > > > > >> > >> >> > > should > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I think > > > doing > > > > > > > checking > > > > > > > > >> on > > > > > > > > >> > the > > > > > > > > >> > >> >> > client > > > > > > > > >> > >> >> > > > side > > > > > > > > >> > >> >> > > > > > is > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we > can > > > > > > discover > > > > > > > > the > > > > > > > > >> > >> problem > > > > > > > > >> > >> >> > > before > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > always > > a > > > > good > > > > > > > > thing. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But we > can > > > not > > > > > > only > > > > > > > > >> rely on > > > > > > > > >> > >> the > > > > > > > > >> > >> >> > > client > > > > > > > > >> > >> >> > > > > side > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > cluster > > > > > > > > >> > >> TaskManagers > > > > > > > > >> > >> >> on > > > > > > > > >> > >> >> > > > > > different > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > different > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > and > > > > > > > the > > > > > > > > >> > client > > > > > > > > >> > >> >> does > > > > > > > > >> > >> >> > > see > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What do > > you > > > > > think? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thank > you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong > > Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On Thu, > > Aug > > > 8, > > > > > > 2019 > > > > > > > at > > > > > > > > >> 5:09 > > > > > > > > >> > >> PM > > > > > > > > >> > >> >> Yang > > > > > > > > >> > >> >> > > > Wang > > > > > > > > >> > >> >> > > > > < > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > > xintong, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > > your > > > > > > > > >> detailed > > > > > > > > >> > >> >> > proposal. > > > > > > > > >> > >> >> > > > > After > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > it > > > > > > > will > > > > > > > > be > > > > > > > > >> > more > > > > > > > > >> > >> >> > > powerful > > > > > > > > >> > >> >> > > > to > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > just > > have > > > > few > > > > > > > > >> questions > > > > > > > > >> > >> about > > > > > > > > >> > >> >> it. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > Native > > > > > and > > > > > > > > Direct > > > > > > > > >> > >> Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We do > > not > > > > > > > > >> differentiate > > > > > > > > >> > >> user > > > > > > > > >> > >> >> > direct > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > included > > > in > > > > > task > > > > > > > > >> off-heap > > > > > > > > >> > >> >> memory. > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > Memory > > > > > > > > >> Calculation > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If the > > sum > > > > of > > > > > > and > > > > > > > > >> > >> fine-grained > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > larger > > > > than > > > > > > > total > > > > > > > > >> > >> process > > > > > > > > >> > >> >> > > memory, > > > > > > > > >> > >> >> > > > > how > > > > > > > > >> > >> >> > > > > > do > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we > need > > to > > > > > check > > > > > > > the > > > > > > > > >> > memory > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song < > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > everyone, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > > would > > > > > like > > > > > > to > > > > > > > > >> start > > > > > > > > >> > a > > > > > > > > >> > >> >> > > discussion > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > for > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > memory > > > > > > > > >> > >> >> > > configurations. > > > > > > > > >> > >> >> > > > > The > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > early > > > > design > > > > > > > > "Memory > > > > > > > > >> > >> >> Management > > > > > > > > >> > >> >> > > and > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > with > > > > updates > > > > > > > from > > > > > > > > >> > >> follow-up > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > This > > > FLIP > > > > > > > > addresses > > > > > > > > >> > >> several > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > memory > > > > > > > > >> > >> >> > > configuration. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Different > > > > > > > > >> > >> configuration > > > > > > > > >> > >> >> > for > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Complex > > > > > > and > > > > > > > > >> > >> difficult > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Complicated, > > > > > > > > >> > >> uncertain > > > > > > > > >> > >> >> and > > > > > > > > >> > >> >> > > > hard > > > > > > > > >> > >> >> > > > > to > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Key > > > > changes > > > > > to > > > > > > > > solve > > > > > > > > >> > the > > > > > > > > >> > >> >> > problems > > > > > > > > >> > >> >> > > > can > > > > > > > > >> > >> >> > > > > > be > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Extend > > > > > > > memory > > > > > > > > >> > >> manager > > > > > > > > >> > >> >> to > > > > > > > > >> > >> >> > > also > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > Modify > > > > > > how > > > > > > > > >> > >> TaskExecutor > > > > > > > > >> > >> >> > > memory > > > > > > > > >> > >> >> > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > >> reservations > > > > > > > > >> > >> and > > > > > > > > >> > >> >> > pools. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Simplify > > > > > > > > memory > > > > > > > > >> > >> >> > > configuration > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > find > > > > > > more > > > > > > > > >> > details > > > > > > > > >> > >> in > > > > > > > > >> > >> >> the > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > note > > > > > > > that > > > > > > > > >> the > > > > > > > > >> > >> early > > > > > > > > >> > >> >> > > design > > > > > > > > >> > >> >> > > > > doc > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > to > > > > > > > > have > > > > > > > > >> the > > > > > > > > >> > >> >> > > discussion > > > > > > > > >> > >> >> > > > in > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > forward > > > > > > > to > > > > > > > > >> your > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > you~ > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
|
|
@Till
Just trying to understand. Are you suggesting that we have both a task executor level memory manager, and multiple slot level memory managers with in the task executor level one? (Or what ever the components are called) If that is what you mean, then it makes a lot sense to me. Thank you~ Xintong Song On Mon, Sep 16, 2019 at 5:30 PM Till Rohrmann <[hidden email]> wrote: > I think conceptually it makes sense to have a MemoryManager per slot. As > Andrey said, it should not add any noticeable overhead. Moreover, I think > this model does not contradict any of the mentioned future improvements. > For example, a per slot MemoryManager could still be implemented such that > the TaskExecutor's MemoryManager keeps a reference to it and can request to > give up/assign more memory segments in case of revocable memory. The > benefit I see is a better abstraction and hence a more targeted component > which will be easier to maintain and evolve. > > Cheers, > Till > > On Mon, Sep 16, 2019 at 11:23 AM Xintong Song <[hidden email]> > wrote: > > > I would say both ways (one unified memory manager per task executor, or > one > > isolated memory manager per slot) should work well at the moment. I've > not > > yet think this through very detailedly, but just from what I can think of > > at the moment, I'm in favor of having one memory manager per task > executor, > > for the following reasons: > > > > - We need the memory manager to understand quota of each operator anyway > > (with the provided fractions), for limiting the memory usage of > > un-revocable operators. It shouldn't be much more complicated for the > > memory manager to isolate memory across slots. Actually I'm wondering > > wether it requires more efforts at all compare to having one memory > manager > > per slot, where we also need to handle the creation and destroying of > > memory managers. > > > > - It is true that we have no plan for segments caching at the moment. > But I > > would say it is a good opportunity for future improvement. E.g., we may > > have some "lazy de-allocation" strategy that only de-allocate segments if > > the remaining available memory quota cannot satisfy new coming > reservation > > requests. I understand that we should not complicate the current design > for > > any "potential future features". My concern is that we make efforts > > changing from one memory manager per task executor to one per slot, with > > the benefit (easier to isolate memory across slots) that is uncertain to > > me, and later discover that we may want to change it back for further > > optimizations. > > > > Anyway, I do not have strong objection against having one memory manager > > per slot. Just raising my concerns. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Sep 16, 2019 at 4:16 PM Andrey Zagrebin <[hidden email]> > > wrote: > > > > > @Xintong > > > > > > Just to understand, what kind of overhead do you mean by creating the > > > memory manager for every dynamic slot? > > > We agreed not to preallocate segments anymore and there was no plan to > do > > > any segment caching at the moment, AFAIK. > > > This means the segments are just garbage collected upon release and not > > > reused. > > > If we decide to go with caching or revoking later and share pools among > > > slots, it would already mean mixed segment/reservation types of memory > > > allocation in general (like batch and streaming). > > > Then I think memory manager would need some input from tasks because it > > is > > > hard for Flink to decide how much to cache and which pool to allocate > > from > > > eventually. > > > As I understand, tasks do not really need this now. That is why I think > > > having memory manager per slot will give more security by isolating > slots > > > with relatively small effort. > > > If there is no preallocation/caching then creating the memory manager > is > > > almost no-op, later we can reconsider it if needed. What do you think? > > > > > > Thanks, > > > Andrey > > > > > > On Sat, Sep 14, 2019 at 8:11 AM Xintong Song <[hidden email]> > > > wrote: > > > > > > > @Andrey, > > > > > > > > If we speak in the condition of dynamic slot allocation, then DataSet > > > tasks > > > > should always request slots with unknown resource profiles. Then the > > > > allocated slots should have 1/n on-heap managed memory and 1/n > off-heap > > > > managed memory of the task executor, where n is the configured > > > > 'numberOfSlot'. Given that DataSet operators only allocate segments > (no > > > > reservation), and does not care whether the segments are on-heap or > > > > off-heap, I think it makes sense to wrapping both pools for DataSet > > > > operators, with respect to the on-heap/off-heap quota of the > allocated > > > > slot. > > > > > > > > I'm not sure whether we want to create one memory manager per slot. I > > > think > > > > we can have limit the quota of each slot with one memory manager per > > task > > > > executor. Given the dynamic slot allocation that dynamically create > and > > > > destroy slots, I think having one memory manager per task executor > may > > > save > > > > the overhead for frequently creating and destroying memory managers, > > and > > > > provide the chance for reusing memory segments across slots. > > > > > > > > Thank you~ > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 11:47 PM Andrey Zagrebin < > [hidden email] > > > > > > > wrote: > > > > > > > > > Hi Xintong, > > > > > > > > > > True, there would be no regression if only one type of memory is > > > > > configured. This can be a problem only for the old jobs running in > a > > > > newly > > > > > configured cluster. > > > > > > > > > > About the pool type precedence, in general, it should not matter > for > > > the > > > > > users which type the segments have. > > > > > The first implementation can be just to pull from any pool, e.g. > > empty > > > > one > > > > > pool firstly and then another or some other random pulling. > > > > > This might be a problem if we mix segment allocations and > reservation > > > of > > > > > memory chunks from the same memory manager. > > > > > The reservation will be usually for a certain type of memory then > the > > > > task > > > > > will probably have to also decide from which pool to allocate the > > > > segments. > > > > > I would suggest we create a memory manager per slot and give it > > memory > > > > > limit of the slot then we do not have this kind of mixed operation > > > > > because Dataset/Batch jobs need only segment memory allocations and > > > > > streaming jobs need only memory chunks for state backends as I > > > understand > > > > > the current plan. > > > > > I would suggest we will look at it if we have the mixed operations > at > > > > some > > > > > point and it becomes a problem. > > > > > > > > > > Thanks, > > > > > Andrey > > > > > > > > > > On Fri, Sep 13, 2019 at 5:24 PM Andrey Zagrebin < > > [hidden email]> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > From: Xintong Song <[hidden email]> > > > > > > Date: Thu, Sep 12, 2019 at 4:21 AM > > > > > > Subject: Re: [DISCUSS] FLIP-49: Unified Memory Configuration for > > > > > > TaskExecutors > > > > > > To: dev <[hidden email]> > > > > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > > > Thanks for bringing this up. > > > > > > > > > > > > If I understand correctly, this issue only occurs where the > cluster > > > is > > > > > > configured with both on-heap and off-heap memory. There should be > > no > > > > > > regression for clusters configured in the old way (either all > > on-heap > > > > or > > > > > > all off-heap). > > > > > > > > > > > > I also agree that it would be good if the DataSet API jobs can > use > > > both > > > > > > memory types. The only question I can see is that, from which > pool > > > > (heap > > > > > / > > > > > > off-heap) should we allocate memory for DataSet API operators? Do > > we > > > > > > always prioritize one pool over the other? Or do we always > > prioritize > > > > the > > > > > > pool with more available memory left? > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 8:15 PM Andrey Zagrebin < > > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > While looking more into the implementation details of Step 4, > we > > > > > released > > > > > > > during some offline discussions with @Till > > > > > > > that there can be a performance degradation for the batch > DataSet > > > API > > > > > if > > > > > > we > > > > > > > simply continue to pull memory from the pool > > > > > > > according the legacy option taskmanager.memory.off-heap. > > > > > > > > > > > > > > The reason is that if the cluster is newly configured to > > statically > > > > > split > > > > > > > heap/off-heap (not like previously either heap or 0ff-heap) > > > > > > > then the batch DataSet API jobs will be able to use only one > type > > > of > > > > > > > memory. Although it does not really matter where the memory > > > segments > > > > > come > > > > > > > from > > > > > > > and potentially batch jobs can use both. Also, currently the > > > Dataset > > > > > API > > > > > > > does not result in absolute resource requirements and its batch > > > jobs > > > > > will > > > > > > > always get a default share of TM resources. > > > > > > > > > > > > > > The suggestion is that we let the batch tasks of Dataset API > pull > > > > from > > > > > > both > > > > > > > pools according to their fair slot share of each memory type. > > > > > > > For that we can have a special wrapping view of both pools > which > > > will > > > > > > pull > > > > > > > segments (can be randomly) according to the slot limits. > > > > > > > The view can wrap TM level memory pools and be given to the > Task. > > > > > > > > > > > > > > Best, > > > > > > > Andrey > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:35 PM Xintong Song < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for your comments, Andrey. > > > > > > > > > > > > > > > > - Regarding Task Off-Heap Memory, I think you're right that > the > > > > user > > > > > > need > > > > > > > > to make sure that direct memory and native memory together > used > > > by > > > > > the > > > > > > > user > > > > > > > > code (external libs) do not exceed the configured value. As > far > > > as > > > > I > > > > > > can > > > > > > > > think of, there is nothing we can do about it. > > > > > > > > > > > > > > > > I addressed the rest of your comment in the wiki page [1]. > > Please > > > > > take > > > > > > a > > > > > > > > look. > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 6:13 PM Andrey Zagrebin < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > EDIT: sorry for confusion I meant > > > > > > > > > taskmanager.memory.off-heap > > > > > > > > > instead of > > > > > > > > > setting taskmanager.memory.preallocate > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 11:29 AM Andrey Zagrebin < > > > > > > [hidden email]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > @Xitong thanks a lot for driving the discussion. > > > > > > > > > > > > > > > > > > > > I also reviewed the FLIP and it looks quite good to me. > > > > > > > > > > Here are some comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I wanted to discuss is the > > > > backwards-compatibility > > > > > > > with > > > > > > > > > > the previous user setups. We could list which options > we > > > > plan > > > > > to > > > > > > > > > deprecate. > > > > > > > > > > From the first glance it looks possible to provide the > > > > > > > same/similar > > > > > > > > > > behaviour for the setups relying on the deprecated > > > options. > > > > > E.g. > > > > > > > > > > setting taskmanager.memory.preallocate to true could > > > > override > > > > > > the > > > > > > > > > > new taskmanager.memory.managed.offheap-fraction to 1 > > etc. > > > At > > > > > the > > > > > > > > > moment the > > > > > > > > > > FLIP just states that in some cases it may require > > > > > > re-configuring > > > > > > > of > > > > > > > > > > cluster if migrated from prior versions. My suggestion > > is > > > > that > > > > > > we > > > > > > > > try > > > > > > > > > to > > > > > > > > > > keep it backwards-compatible unless there is a good > > reason > > > > > like > > > > > > > some > > > > > > > > > major > > > > > > > > > > complication for the implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also couple of smaller things: > > > > > > > > > > > > > > > > > > > > - I suggest we remove TaskExecutorSpecifics from the > > FLIP > > > > and > > > > > > > leave > > > > > > > > > > some general wording atm, like 'data structure to > store' > > > or > > > > > > > 'utility > > > > > > > > > > classes'. When the classes are implemented, we put the > > > > > concrete > > > > > > > > class > > > > > > > > > > names. This way we can avoid confusion and stale > > > documents. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - As I understand, if user task uses native memory > (not > > > > direct > > > > > > > > memory, > > > > > > > > > > but e.g. unsafe.allocate or from external lib), there > > will > > > > be > > > > > no > > > > > > > > > > explicit guard against exceeding 'task off heap > memory'. > > > > Then > > > > > > user > > > > > > > > > should > > > > > > > > > > still explicitly make sure that her/his direct buffer > > > > > allocation > > > > > > > > plus > > > > > > > > > any > > > > > > > > > > other memory usages does not exceed value announced as > > > 'task > > > > > off > > > > > > > > > heap'. I > > > > > > > > > > guess there is no so much that can be done about it > > except > > > > > > > > mentioning > > > > > > > > > in > > > > > > > > > > docs, similar to controlling the heap state backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 10:07 AM Yang Wang < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> I also agree that all the configuration should be > > calculated > > > > out > > > > > > of > > > > > > > > > >> TaskManager. > > > > > > > > > >> > > > > > > > > > >> So a full configuration should be generated before > > > TaskManager > > > > > > > > started. > > > > > > > > > >> > > > > > > > > > >> Override the calculated configurations through -D now > > seems > > > > > > better. > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> Best, > > > > > > > > > >> > > > > > > > > > >> Yang > > > > > > > > > >> > > > > > > > > > >> Xintong Song <[hidden email]> 于2019年9月2日周一 > > > 上午11:39写道： > > > > > > > > > >> > > > > > > > > > >> > I just updated the FLIP wiki page [1], with the > > following > > > > > > changes: > > > > > > > > > >> > > > > > > > > > > >> > - Network memory uses JVM direct memory, and is > > > accounted > > > > > > when > > > > > > > > > >> setting > > > > > > > > > >> > JVM max direct memory size parameter. > > > > > > > > > >> > - Use dynamic configurations (`-Dkey=value`) to > pass > > > > > > calculated > > > > > > > > > >> memory > > > > > > > > > >> > configs into TaskExecutors, instead of ENV > variables. > > > > > > > > > >> > - Remove 'supporting memory reservation' from the > > scope > > > > of > > > > > > this > > > > > > > > > FLIP. > > > > > > > > > >> > > > > > > > > > > >> > @till @stephan, please take another look see if there > > are > > > > any > > > > > > > other > > > > > > > > > >> > concerns. > > > > > > > > > >> > > > > > > > > > > >> > Thank you~ > > > > > > > > > >> > > > > > > > > > > >> > Xintong Song > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > [1] > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > >> > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > >> > > > > > > > > > > >> > > Sorry for the late response. > > > > > > > > > >> > > > > > > > > > > > >> > > - Regarding the `TaskExecutorSpecifics` naming, > let's > > > > > discuss > > > > > > > the > > > > > > > > > >> detail > > > > > > > > > >> > > in PR. > > > > > > > > > >> > > - Regarding passing parameters into the > > `TaskExecutor`, > > > +1 > > > > > for > > > > > > > > using > > > > > > > > > >> > > dynamic configuration at the moment, given that > there > > > are > > > > > more > > > > > > > > > >> questions > > > > > > > > > >> > to > > > > > > > > > >> > > be discussed to have a general framework for > > overwriting > > > > > > > > > >> configurations > > > > > > > > > >> > > with ENV variables. > > > > > > > > > >> > > - Regarding memory reservation, I double checked > with > > Yu > > > > and > > > > > > he > > > > > > > > will > > > > > > > > > >> take > > > > > > > > > >> > > care of it. > > > > > > > > > >> > > > > > > > > > > > >> > > Thank you~ > > > > > > > > > >> > > > > > > > > > > > >> > > Xintong Song > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > >> > > > > > > > > > > > >> > >> What I forgot to add is that we could tackle > > specifying > > > > the > > > > > > > > > >> > configuration > > > > > > > > > >> > >> fully in an incremental way and that the full > > > > specification > > > > > > > > should > > > > > > > > > be > > > > > > > > > >> > the > > > > > > > > > >> > >> desired end state. > > > > > > > > > >> > >> > > > > > > > > > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann < > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> wrote: > > > > > > > > > >> > >> > > > > > > > > > >> > >> > I think our goal should be that the configuration > > is > > > > > fully > > > > > > > > > >> specified > > > > > > > > > >> > >> when > > > > > > > > > >> > >> > the process is started. By considering the > internal > > > > > > > calculation > > > > > > > > > >> step > > > > > > > > > >> > to > > > > > > > > > >> > >> be > > > > > > > > > >> > >> > rather validate existing values and calculate > > missing > > > > > ones, > > > > > > > > these > > > > > > > > > >> two > > > > > > > > > >> > >> > proposal shouldn't even conflict (given > > determinism). > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > Since we don't want to change an existing > > > > > flink-conf.yaml, > > > > > > > > > >> specifying > > > > > > > > > >> > >> the > > > > > > > > > >> > >> > full configuration would require to pass in the > > > options > > > > > > > > > >> differently. > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > One way could be the ENV variables approach. The > > > reason > > > > > why > > > > > > > I'm > > > > > > > > > >> trying > > > > > > > > > >> > >> to > > > > > > > > > >> > >> > exclude this feature from the FLIP is that I > > believe > > > it > > > > > > > needs a > > > > > > > > > bit > > > > > > > > > >> > more > > > > > > > > > >> > >> > discussion. Just some questions which come to my > > > mind: > > > > > What > > > > > > > > would > > > > > > > > > >> be > > > > > > > > > >> > the > > > > > > > > > >> > >> > exact format (FLINK_KEY_NAME)? Would we support a > > dot > > > > > > > separator > > > > > > > > > >> which > > > > > > > > > >> > is > > > > > > > > > >> > >> > supported by some systems (FLINK.KEY.NAME)? If > we > > > > accept > > > > > > the > > > > > > > > dot > > > > > > > > > >> > >> > separator what would be the order of precedence > if > > > > there > > > > > > are > > > > > > > > two > > > > > > > > > >> ENV > > > > > > > > > >> > >> > variables defined (FLINK_KEY_NAME and > > FLINK.KEY.NAME > > > )? > > > > > > What > > > > > > > is > > > > > > > > > the > > > > > > > > > >> > >> > precedence of env variable vs. dynamic > > configuration > > > > > value > > > > > > > > > >> specified > > > > > > > > > >> > >> via -D? > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > Another approach could be to pass in the dynamic > > > > > > > configuration > > > > > > > > > >> values > > > > > > > > > >> > >> via > > > > > > > > > >> > >> > `-Dkey=value` to the Flink process. For that we > > don't > > > > > have > > > > > > to > > > > > > > > > >> change > > > > > > > > > >> > >> > anything because the functionality already > exists. > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > Cheers, > > > > > > > > > >> > >> > Till > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan Ewen < > > > > > > > > [hidden email]> > > > > > > > > > >> > wrote: > > > > > > > > > >> > >> > > > > > > > > > > >> > >> >> I see. Under the assumption of strict > determinism > > > that > > > > > > > should > > > > > > > > > >> work. > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> The original proposal had this point "don't > > compute > > > > > inside > > > > > > > the > > > > > > > > > TM, > > > > > > > > > >> > >> compute > > > > > > > > > >> > >> >> outside and supply a full config", because that > > > > sounded > > > > > > more > > > > > > > > > >> > intuitive. > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till Rohrmann < > > > > > > > > > >> [hidden email] > > > > > > > > > >> > > > > > > > > > > > >> > >> >> wrote: > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> >> > My understanding was that before starting the > > > Flink > > > > > > > process > > > > > > > > we > > > > > > > > > >> > call a > > > > > > > > > >> > >> >> > utility which calculates these values. I > assume > > > that > > > > > > this > > > > > > > > > >> utility > > > > > > > > > >> > >> will > > > > > > > > > >> > >> >> do > > > > > > > > > >> > >> >> > the calculation based on a set of configured > > > values > > > > > > > (process > > > > > > > > > >> > memory, > > > > > > > > > >> > >> >> flink > > > > > > > > > >> > >> >> > memory, network memory etc.). Assuming that > > these > > > > > values > > > > > > > > don't > > > > > > > > > >> > differ > > > > > > > > > >> > >> >> from > > > > > > > > > >> > >> >> > the values with which the JVM is started, it > > > should > > > > be > > > > > > > > > possible > > > > > > > > > >> to > > > > > > > > > >> > >> >> > recompute them in the Flink process in order > to > > > set > > > > > the > > > > > > > > > values. > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan Ewen > < > > > > > > > > > [hidden email] > > > > > > > > > >> > > > > > > > > > > >> > >> wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > When computing the values in the JVM process > > > after > > > > > it > > > > > > > > > started, > > > > > > > > > >> > how > > > > > > > > > >> > >> >> would > > > > > > > > > >> > >> >> > > you deal with values like Max Direct Memory, > > > > > Metaspace > > > > > > > > size. > > > > > > > > > >> > native > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > reservation (reduce heap size), etc? All the > > > > values > > > > > > that > > > > > > > > are > > > > > > > > > >> > >> >> parameters > > > > > > > > > >> > >> >> > to > > > > > > > > > >> > >> >> > > the JVM process and that need to be supplied > > at > > > > > > process > > > > > > > > > >> startup? > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till > Rohrmann > > < > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > Thanks for the clarification. I have some > > more > > > > > > > comments: > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > - I would actually split the logic to > > compute > > > > the > > > > > > > > process > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > > requirements and storing the values into > two > > > > > things. > > > > > > > > E.g. > > > > > > > > > >> one > > > > > > > > > >> > >> could > > > > > > > > > >> > >> >> > name > > > > > > > > > >> > >> >> > > > the former TaskExecutorProcessUtility and > > the > > > > > > latter > > > > > > > > > >> > >> >> > > > TaskExecutorProcessMemory. But we can > > discuss > > > > this > > > > > > on > > > > > > > > the > > > > > > > > > PR > > > > > > > > > >> > >> since > > > > > > > > > >> > >> >> it's > > > > > > > > > >> > >> >> > > > just a naming detail. > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > - Generally, I'm not opposed to making > > > > > configuration > > > > > > > > > values > > > > > > > > > >> > >> >> overridable > > > > > > > > > >> > >> >> > > by > > > > > > > > > >> > >> >> > > > ENV variables. I think this is a very good > > > idea > > > > > and > > > > > > > > makes > > > > > > > > > >> the > > > > > > > > > >> > >> >> > > > configurability of Flink processes easier. > > > > > However, > > > > > > I > > > > > > > > > think > > > > > > > > > >> > that > > > > > > > > > >> > >> >> adding > > > > > > > > > >> > >> >> > > > this functionality should not be part of > > this > > > > FLIP > > > > > > > > because > > > > > > > > > >> it > > > > > > > > > >> > >> would > > > > > > > > > >> > >> >> > > simply > > > > > > > > > >> > >> >> > > > widen the scope unnecessarily. > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > The reasons why I believe it is > unnecessary > > > are > > > > > the > > > > > > > > > >> following: > > > > > > > > > >> > >> For > > > > > > > > > >> > >> >> Yarn > > > > > > > > > >> > >> >> > > we > > > > > > > > > >> > >> >> > > > already create write a flink-conf.yaml > which > > > > could > > > > > > be > > > > > > > > > >> populated > > > > > > > > > >> > >> with > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > memory settings. For the other processes > it > > > > should > > > > > > not > > > > > > > > > make > > > > > > > > > >> a > > > > > > > > > >> > >> >> > difference > > > > > > > > > >> > >> >> > > > whether the loaded Configuration is > > populated > > > > with > > > > > > the > > > > > > > > > >> memory > > > > > > > > > >> > >> >> settings > > > > > > > > > >> > >> >> > > from > > > > > > > > > >> > >> >> > > > ENV variables or by using > > > > > TaskExecutorProcessUtility > > > > > > > to > > > > > > > > > >> compute > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > missing > > > > > > > > > >> > >> >> > > > values from the loaded configuration. If > the > > > > > latter > > > > > > > > would > > > > > > > > > >> not > > > > > > > > > >> > be > > > > > > > > > >> > >> >> > possible > > > > > > > > > >> > >> >> > > > (wrong or missing configuration values), > > then > > > we > > > > > > > should > > > > > > > > > not > > > > > > > > > >> > have > > > > > > > > > >> > >> >> been > > > > > > > > > >> > >> >> > > able > > > > > > > > > >> > >> >> > > > to actually start the process in the first > > > > place. > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > - Concerning the memory reservation: I > agree > > > > with > > > > > > you > > > > > > > > that > > > > > > > > > >> we > > > > > > > > > >> > >> need > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > memory reservation functionality to make > > > > streaming > > > > > > > jobs > > > > > > > > > work > > > > > > > > > >> > with > > > > > > > > > >> > >> >> > > "managed" > > > > > > > > > >> > >> >> > > > memory. However, w/o this functionality > the > > > > whole > > > > > > Flip > > > > > > > > > would > > > > > > > > > >> > >> already > > > > > > > > > >> > >> >> > > bring > > > > > > > > > >> > >> >> > > > a good amount of improvements to our users > > > when > > > > > > > running > > > > > > > > > >> batch > > > > > > > > > >> > >> jobs. > > > > > > > > > >> > >> >> > > > Moreover, by keeping the scope smaller we > > can > > > > > > complete > > > > > > > > the > > > > > > > > > >> FLIP > > > > > > > > > >> > >> >> faster. > > > > > > > > > >> > >> >> > > > Hence, I would propose to address the > memory > > > > > > > reservation > > > > > > > > > >> > >> >> functionality > > > > > > > > > >> > >> >> > > as a > > > > > > > > > >> > >> >> > > > follow up FLIP (which Yu is working on if > > I'm > > > > not > > > > > > > > > mistaken). > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > Cheers, > > > > > > > > > >> > >> >> > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang > Wang < > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > Just add my 2 cents. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > Using environment variables to override > > the > > > > > > > > > configuration > > > > > > > > > >> for > > > > > > > > > >> > >> >> > different > > > > > > > > > >> > >> >> > > > > taskmanagers is better. > > > > > > > > > >> > >> >> > > > > We do not need to generate dedicated > > > > > > flink-conf.yaml > > > > > > > > for > > > > > > > > > >> all > > > > > > > > > >> > >> >> > > > taskmanagers. > > > > > > > > > >> > >> >> > > > > A common flink-conf.yam and different > > > > > environment > > > > > > > > > >> variables > > > > > > > > > >> > are > > > > > > > > > >> > >> >> > enough. > > > > > > > > > >> > >> >> > > > > By reducing the distributed cached > files, > > it > > > > > could > > > > > > > > make > > > > > > > > > >> > >> launching > > > > > > > > > >> > >> >> a > > > > > > > > > >> > >> >> > > > > taskmanager faster. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > Stephan gives a good suggestion that we > > > could > > > > > move > > > > > > > the > > > > > > > > > >> logic > > > > > > > > > >> > >> into > > > > > > > > > >> > >> >> > > > > "GlobalConfiguration.loadConfig()" > method. > > > > > > > > > >> > >> >> > > > > Maybe the client could also benefit from > > > this. > > > > > > > > Different > > > > > > > > > >> > users > > > > > > > > > >> > >> do > > > > > > > > > >> > >> >> not > > > > > > > > > >> > >> >> > > > have > > > > > > > > > >> > >> >> > > > > to export FLINK_CONF_DIR to update few > > > config > > > > > > > options. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > Best, > > > > > > > > > >> > >> >> > > > > Yang > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > Stephan Ewen <[hidden email]> > > > 于2019年8月28日周三 > > > > > > > > 上午1:21写道： > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > One note on the Environment Variables > > and > > > > > > > > > Configuration > > > > > > > > > >> > >> >> discussion. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > My understanding is that passed ENV > > > > variables > > > > > > are > > > > > > > > > added > > > > > > > > > >> to > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > > > > configuration in the > > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > >> > >> method > > > > > > > > > >> > >> >> (or > > > > > > > > > >> > >> >> > > > > > similar). > > > > > > > > > >> > >> >> > > > > > For all the code inside Flink, it > looks > > > like > > > > > the > > > > > > > > data > > > > > > > > > >> was > > > > > > > > > >> > in > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > config > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > start with, just that the scripts that > > > > compute > > > > > > the > > > > > > > > > >> > variables > > > > > > > > > >> > >> can > > > > > > > > > >> > >> >> > pass > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > values to the process without actually > > > > needing > > > > > > to > > > > > > > > > write > > > > > > > > > >> a > > > > > > > > > >> > >> file. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > For example the > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > >> method > > > > > > > > > >> > >> would > > > > > > > > > >> > >> >> > take > > > > > > > > > >> > >> >> > > > any > > > > > > > > > >> > >> >> > > > > > ENV variable prefixed with "flink" and > > add > > > > it > > > > > > as a > > > > > > > > > >> config > > > > > > > > > >> > >> key. > > > > > > > > > >> > >> >> > > > > > "flink_taskmanager_memory_size=2g" > would > > > > > become > > > > > > > > > >> > >> >> > > > "taskmanager.memory.size: > > > > > > > > > >> > >> >> > > > > > 2g". > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM > Xintong > > > > Song < > > > > > > > > > >> > >> >> > [hidden email]> > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > Thanks for the comments, Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > I've also seen your comments on the > > wiki > > > > > page, > > > > > > > but > > > > > > > > > >> let's > > > > > > > > > >> > >> keep > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > > > > discussion here. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding 'TaskExecutorSpecifics', > > how > > > > do > > > > > > you > > > > > > > > > think > > > > > > > > > >> > about > > > > > > > > > >> > >> >> > naming > > > > > > > > > >> > >> >> > > it > > > > > > > > > >> > >> >> > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > > > > >> > >> >> > > > > > > - Regarding passing memory > > > configurations > > > > > into > > > > > > > > task > > > > > > > > > >> > >> executors, > > > > > > > > > >> > >> >> > I'm > > > > > > > > > >> > >> >> > > in > > > > > > > > > >> > >> >> > > > > > favor > > > > > > > > > >> > >> >> > > > > > > of do it via environment variables > > > rather > > > > > than > > > > > > > > > >> > >> configurations, > > > > > > > > > >> > >> >> > with > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > following two reasons. > > > > > > > > > >> > >> >> > > > > > > - It is easier to keep the memory > > > > options > > > > > > once > > > > > > > > > >> > calculate > > > > > > > > > >> > >> >> not to > > > > > > > > > >> > >> >> > > be > > > > > > > > > >> > >> >> > > > > > > changed with environment variables > > > rather > > > > > than > > > > > > > > > >> > >> configurations. > > > > > > > > > >> > >> >> > > > > > > - I'm not sure whether we should > > write > > > > the > > > > > > > > > >> > configuration > > > > > > > > > >> > >> in > > > > > > > > > >> > >> >> > > startup > > > > > > > > > >> > >> >> > > > > > > scripts. Writing changes into the > > > > > > configuration > > > > > > > > > files > > > > > > > > > >> > when > > > > > > > > > >> > >> >> > running > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > startup scripts does not sounds > right > > to > > > > me. > > > > > > Or > > > > > > > we > > > > > > > > > >> could > > > > > > > > > >> > >> make > > > > > > > > > >> > >> >> a > > > > > > > > > >> > >> >> > > copy > > > > > > > > > >> > >> >> > > > of > > > > > > > > > >> > >> >> > > > > > > configuration files per flink > cluster, > > > and > > > > > > make > > > > > > > > the > > > > > > > > > >> task > > > > > > > > > >> > >> >> executor > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > load > > > > > > > > > >> > >> >> > > > > > > from the copy, and clean up the copy > > > after > > > > > the > > > > > > > > > >> cluster is > > > > > > > > > >> > >> >> > shutdown, > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > >> > >> >> > > > > > > is complicated. (I think this is > also > > > what > > > > > > > Stephan > > > > > > > > > >> means > > > > > > > > > >> > in > > > > > > > > > >> > >> >> his > > > > > > > > > >> > >> >> > > > comment > > > > > > > > > >> > >> >> > > > > > on > > > > > > > > > >> > >> >> > > > > > > the wiki page?) > > > > > > > > > >> > >> >> > > > > > > - Regarding reserving memory, I > think > > > this > > > > > > > change > > > > > > > > > >> should > > > > > > > > > >> > be > > > > > > > > > >> > >> >> > > included > > > > > > > > > >> > >> >> > > > in > > > > > > > > > >> > >> >> > > > > > > this FLIP. I think a big part of > > > > motivations > > > > > > of > > > > > > > > this > > > > > > > > > >> FLIP > > > > > > > > > >> > >> is > > > > > > > > > >> > >> >> to > > > > > > > > > >> > >> >> > > unify > > > > > > > > > >> > >> >> > > > > > > memory configuration for streaming / > > > batch > > > > > and > > > > > > > > make > > > > > > > > > it > > > > > > > > > >> > easy > > > > > > > > > >> > >> >> for > > > > > > > > > >> > >> >> > > > > > configuring > > > > > > > > > >> > >> >> > > > > > > rocksdb memory. If we don't support > > > memory > > > > > > > > > >> reservation, > > > > > > > > > >> > >> then > > > > > > > > > >> > >> >> > > > streaming > > > > > > > > > >> > >> >> > > > > > jobs > > > > > > > > > >> > >> >> > > > > > > cannot use managed memory (neither > > > on-heap > > > > > or > > > > > > > > > >> off-heap), > > > > > > > > > >> > >> which > > > > > > > > > >> > >> >> > > makes > > > > > > > > > >> > >> >> > > > > this > > > > > > > > > >> > >> >> > > > > > > FLIP incomplete. > > > > > > > > > >> > >> >> > > > > > > - Regarding network memory, I think > > you > > > > are > > > > > > > > right. I > > > > > > > > > >> > think > > > > > > > > > >> > >> we > > > > > > > > > >> > >> >> > > > probably > > > > > > > > > >> > >> >> > > > > > > don't need to change network stack > > from > > > > > using > > > > > > > > direct > > > > > > > > > >> > >> memory to > > > > > > > > > >> > >> >> > > using > > > > > > > > > >> > >> >> > > > > > unsafe > > > > > > > > > >> > >> >> > > > > > > native memory. Network memory size > is > > > > > > > > deterministic, > > > > > > > > > >> > >> cannot be > > > > > > > > > >> > >> >> > > > reserved > > > > > > > > > >> > >> >> > > > > > as > > > > > > > > > >> > >> >> > > > > > > managed memory does, and cannot be > > > > > overused. I > > > > > > > > think > > > > > > > > > >> it > > > > > > > > > >> > >> also > > > > > > > > > >> > >> >> > works > > > > > > > > > >> > >> >> > > if > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > >> > >> >> > > > > > > simply keep using direct memory for > > > > network > > > > > > and > > > > > > > > > >> include > > > > > > > > > >> > it > > > > > > > > > >> > >> in > > > > > > > > > >> > >> >> jvm > > > > > > > > > >> > >> >> > > max > > > > > > > > > >> > >> >> > > > > > > direct memory size. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM Till > > > > > Rohrmann > > > > > > < > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Hi Xintong, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > thanks for addressing the comments > > and > > > > > > adding > > > > > > > a > > > > > > > > > more > > > > > > > > > >> > >> >> detailed > > > > > > > > > >> > >> >> > > > > > > > implementation plan. I have a > couple > > > of > > > > > > > comments > > > > > > > > > >> > >> concerning > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > > > > > implementation plan: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > - The name `TaskExecutorSpecifics` > > is > > > > not > > > > > > > really > > > > > > > > > >> > >> >> descriptive. > > > > > > > > > >> > >> >> > > > > Choosing > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > >> > >> >> > > > > > > > different name could help here. > > > > > > > > > >> > >> >> > > > > > > > - I'm not sure whether I would > pass > > > the > > > > > > memory > > > > > > > > > >> > >> >> configuration to > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > > > TaskExecutor via environment > > > variables. > > > > I > > > > > > > think > > > > > > > > it > > > > > > > > > >> > would > > > > > > > > > >> > >> be > > > > > > > > > >> > >> >> > > better > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > write > > > > > > > > > >> > >> >> > > > > > > > it into the configuration one uses > > to > > > > > start > > > > > > > the > > > > > > > > TM > > > > > > > > > >> > >> process. > > > > > > > > > >> > >> >> > > > > > > > - If possible, I would exclude the > > > > memory > > > > > > > > > >> reservation > > > > > > > > > >> > >> from > > > > > > > > > >> > >> >> this > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > >> > >> >> > > > > > > > add this as part of a dedicated > > FLIP. > > > > > > > > > >> > >> >> > > > > > > > - If possible, then I would > exclude > > > > > changes > > > > > > to > > > > > > > > the > > > > > > > > > >> > >> network > > > > > > > > > >> > >> >> > stack > > > > > > > > > >> > >> >> > > > from > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > FLIP. Maybe we can simply say that > > the > > > > > > direct > > > > > > > > > memory > > > > > > > > > >> > >> needed > > > > > > > > > >> > >> >> by > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > > network > > > > > > > > > >> > >> >> > > > > > > > stack is the framework direct > memory > > > > > > > > requirement. > > > > > > > > > >> > >> Changing > > > > > > > > > >> > >> >> how > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > is allocated can happen in a > second > > > > step. > > > > > > This > > > > > > > > > would > > > > > > > > > >> > keep > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > scope > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > FLIP smaller. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM > > > Xintong > > > > > > Song < > > > > > > > > > >> > >> >> > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Hi everyone, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > I just updated the FLIP document > > on > > > > wiki > > > > > > > [1], > > > > > > > > > with > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > > following > > > > > > > > > >> > >> >> > > > > > > changes. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > - Removed open question > > regarding > > > > > > > > > MemorySegment > > > > > > > > > >> > >> >> > allocation. > > > > > > > > > >> > >> >> > > As > > > > > > > > > >> > >> >> > > > > > > > > discussed, we exclude this > > topic > > > > from > > > > > > the > > > > > > > > > >> scope of > > > > > > > > > >> > >> this > > > > > > > > > >> > >> >> > > FLIP. > > > > > > > > > >> > >> >> > > > > > > > > - Updated content about JVM > > > direct > > > > > > memory > > > > > > > > > >> > parameter > > > > > > > > > >> > >> >> > > according > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > recent > > > > > > > > > >> > >> >> > > > > > > > > discussions, and moved the > > other > > > > > > options > > > > > > > to > > > > > > > > > >> > >> "Rejected > > > > > > > > > >> > >> >> > > > > > Alternatives" > > > > > > > > > >> > >> >> > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > moment. > > > > > > > > > >> > >> >> > > > > > > > > - Added implementation steps. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > [1] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 PM > > > > Stephan > > > > > > > Ewen < > > > > > > > > > >> > >> >> > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong: Concerning "wait for > > > > memory > > > > > > > users > > > > > > > > > >> before > > > > > > > > > >> > >> task > > > > > > > > > >> > >> >> > > dispose > > > > > > > > > >> > >> >> > > > > and > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > release": I agree, that's how > it > > > > > should > > > > > > > be. > > > > > > > > > >> Let's > > > > > > > > > >> > >> try it > > > > > > > > > >> > >> >> > out. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong @Jingsong: > Concerning " > > > JVM > > > > > > does > > > > > > > > not > > > > > > > > > >> wait > > > > > > > > > >> > >> for > > > > > > > > > >> > >> >> GC > > > > > > > > > >> > >> >> > > when > > > > > > > > > >> > >> >> > > > > > > > allocating > > > > > > > > > >> > >> >> > > > > > > > > > direct memory buffer": There > > seems > > > > to > > > > > be > > > > > > > > > pretty > > > > > > > > > >> > >> >> elaborate > > > > > > > > > >> > >> >> > > logic > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > free > > > > > > > > > >> > >> >> > > > > > > > > > buffers when allocating new > > ones. > > > > See > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Till: Maybe. If we assume > that > > > the > > > > > JVM > > > > > > > > > default > > > > > > > > > >> > works > > > > > > > > > >> > >> >> (like > > > > > > > > > >> > >> >> > > > going > > > > > > > > > >> > >> >> > > > > > > with > > > > > > > > > >> > >> >> > > > > > > > > > option 2 and not setting > > > > > > > > > >> "-XX:MaxDirectMemorySize" > > > > > > > > > >> > at > > > > > > > > > >> > >> >> all), > > > > > > > > > >> > >> >> > > > then > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > >> > >> >> > > > > > > > think > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > > should be okay to set > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > >> to > > > > > > > > > >> > >> >> > > > > > > > > > "off_heap_managed_memory + > > > > > > direct_memory" > > > > > > > > even > > > > > > > > > >> if > > > > > > > > > >> > we > > > > > > > > > >> > >> use > > > > > > > > > >> > >> >> > > > RocksDB. > > > > > > > > > >> > >> >> > > > > > > That > > > > > > > > > >> > >> >> > > > > > > > > is a > > > > > > > > > >> > >> >> > > > > > > > > > big if, though, I honestly > have > > no > > > > > idea > > > > > > :D > > > > > > > > > >> Would be > > > > > > > > > >> > >> >> good to > > > > > > > > > >> > >> >> > > > > > > understand > > > > > > > > > >> > >> >> > > > > > > > > > this, though, because this > would > > > > > affect > > > > > > > > option > > > > > > > > > >> (2) > > > > > > > > > >> > >> and > > > > > > > > > >> > >> >> > option > > > > > > > > > >> > >> >> > > > > > (1.2). > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 > PM > > > > > Xintong > > > > > > > > Song < > > > > > > > > > >> > >> >> > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thanks for the inputs, > > Jingsong. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Let me try to summarize your > > > > points. > > > > > > > > Please > > > > > > > > > >> > correct > > > > > > > > > >> > >> >> me if > > > > > > > > > >> > >> >> > > I'm > > > > > > > > > >> > >> >> > > > > > > wrong. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - Memory consumers should > > > > always > > > > > > > avoid > > > > > > > > > >> > returning > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > > > > segments > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > memory manager while > there > > > are > > > > > > still > > > > > > > > > >> > un-cleaned > > > > > > > > > >> > >> >> > > > structures / > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > may use the memory. > > > Otherwise, > > > > it > > > > > > > would > > > > > > > > > >> cause > > > > > > > > > >> > >> >> serious > > > > > > > > > >> > >> >> > > > > problems > > > > > > > > > >> > >> >> > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > >> > >> >> > > > > > > > > > > multiple consumers trying > > to > > > > use > > > > > > the > > > > > > > > same > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > segment. > > > > > > > > > >> > >> >> > > > > > > > > > > - JVM does not wait for > GC > > > when > > > > > > > > > allocating > > > > > > > > > >> > >> direct > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > > > > buffer. > > > > > > > > > >> > >> >> > > > > > > > > > > Therefore even we set > > proper > > > > max > > > > > > > direct > > > > > > > > > >> memory > > > > > > > > > >> > >> size > > > > > > > > > >> > >> >> > > limit, > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > >> > >> >> > > > > > > may > > > > > > > > > >> > >> >> > > > > > > > > > still > > > > > > > > > >> > >> >> > > > > > > > > > > encounter direct memory > oom > > > if > > > > > the > > > > > > GC > > > > > > > > > >> cleaning > > > > > > > > > >> > >> >> memory > > > > > > > > > >> > >> >> > > > slower > > > > > > > > > >> > >> >> > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > direct memory allocation. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Am I understanding this > > > correctly? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at 4:21 > > PM > > > > > > > > JingsongLee > > > > > > > > > < > > > > > > > > > >> > >> >> > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > .invalid> > > > > > > > > > >> > >> >> > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Hi stephan: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About option 2: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > if additional threads not > > > > cleanly > > > > > > shut > > > > > > > > > down > > > > > > > > > >> > >> before > > > > > > > > > >> > >> >> we > > > > > > > > > >> > >> >> > can > > > > > > > > > >> > >> >> > > > > exit > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > task: > > > > > > > > > >> > >> >> > > > > > > > > > > > In the current case of > > memory > > > > > reuse, > > > > > > > it > > > > > > > > > has > > > > > > > > > >> > >> freed up > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > > > > uses. If this memory is > > used > > > by > > > > > > other > > > > > > > > > tasks > > > > > > > > > >> > and > > > > > > > > > >> > >> >> > > > asynchronous > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > >> > >> >> > > > > > > > > > > > of exited task may still > be > > > > > > writing, > > > > > > > > > there > > > > > > > > > >> > will > > > > > > > > > >> > >> be > > > > > > > > > >> > >> >> > > > > concurrent > > > > > > > > > >> > >> >> > > > > > > > > security > > > > > > > > > >> > >> >> > > > > > > > > > > > problems, and even lead > to > > > > errors > > > > > > in > > > > > > > > user > > > > > > > > > >> > >> computing > > > > > > > > > >> > >> >> > > > results. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > So I think this is a > serious > > > and > > > > > > > > > intolerable > > > > > > > > > >> > >> bug, No > > > > > > > > > >> > >> >> > > matter > > > > > > > > > >> > >> >> > > > > > what > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > option is, it should be > > > > avoided. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About direct memory > cleaned > > by > > > > GC: > > > > > > > > > >> > >> >> > > > > > > > > > > > I don't think it is a good > > > idea, > > > > > > I've > > > > > > > > > >> > >> encountered so > > > > > > > > > >> > >> >> > many > > > > > > > > > >> > >> >> > > > > > > > situations > > > > > > > > > >> > >> >> > > > > > > > > > > > that it's too late for GC > > to > > > > > cause > > > > > > > > > >> > DirectMemory > > > > > > > > > >> > >> >> OOM. > > > > > > > > > >> > >> >> > > > Release > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > allocate DirectMemory > > depend > > > on > > > > > the > > > > > > > > type > > > > > > > > > of > > > > > > > > > >> > user > > > > > > > > > >> > >> >> job, > > > > > > > > > >> > >> >> > > > which > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > often beyond our control. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Best, > > > > > > > > > >> > >> >> > > > > > > > > > > > Jingsong Lee > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > > > > > ------------------------------------------------------------------ > > > > > > > > > >> > >> >> > > > > > > > > > > > From:Stephan Ewen < > > > > > [hidden email] > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Send Time:2019年8月19日(星期一) > > > 15:56 > > > > > > > > > >> > >> >> > > > > > > > > > > > To:dev < > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] > > FLIP-49: > > > > > > Unified > > > > > > > > > >> Memory > > > > > > > > > >> > >> >> > > Configuration > > > > > > > > > >> > >> >> > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > TaskExecutors > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > My main concern with > option > > 2 > > > > > > > (manually > > > > > > > > > >> release > > > > > > > > > >> > >> >> memory) > > > > > > > > > >> > >> >> > > is > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > segfaults > > > > > > > > > >> > >> >> > > > > > > > > > > > in the JVM send off all > > sorts > > > of > > > > > > > alarms > > > > > > > > on > > > > > > > > > >> user > > > > > > > > > >> > >> >> ends. > > > > > > > > > >> > >> >> > So > > > > > > > > > >> > >> >> > > we > > > > > > > > > >> > >> >> > > > > > need > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > guarantee that this never > > > > happens. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > The trickyness is in tasks > > > that > > > > > uses > > > > > > > > data > > > > > > > > > >> > >> >> structures / > > > > > > > > > >> > >> >> > > > > > algorithms > > > > > > > > > >> > >> >> > > > > > > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > additional threads, like > > hash > > > > > table > > > > > > > > > >> spill/read > > > > > > > > > >> > >> and > > > > > > > > > >> > >> >> > > sorting > > > > > > > > > >> > >> >> > > > > > > threads. > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > >> > >> >> > > > > > > > > > > > to ensure that these > cleanly > > > > shut > > > > > > down > > > > > > > > > >> before > > > > > > > > > >> > we > > > > > > > > > >> > >> can > > > > > > > > > >> > >> >> > exit > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > task. > > > > > > > > > >> > >> >> > > > > > > > > > > > I am not sure that we have > > > that > > > > > > > > guaranteed > > > > > > > > > >> > >> already, > > > > > > > > > >> > >> >> > > that's > > > > > > > > > >> > >> >> > > > > why > > > > > > > > > >> > >> >> > > > > > > > option > > > > > > > > > >> > >> >> > > > > > > > > > 1.1 > > > > > > > > > >> > >> >> > > > > > > > > > > > seemed simpler to me. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 at > 3:42 > > > PM > > > > > > > Xintong > > > > > > > > > >> Song < > > > > > > > > > >> > >> >> > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thanks for the comments, > > > > > Stephan. > > > > > > > > > >> Summarized > > > > > > > > > >> > in > > > > > > > > > >> > >> >> this > > > > > > > > > >> > >> >> > > way > > > > > > > > > >> > >> >> > > > > > really > > > > > > > > > >> > >> >> > > > > > > > > makes > > > > > > > > > >> > >> >> > > > > > > > > > > > > things easier to > > understand. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'm in favor of option > 2, > > at > > > > > least > > > > > > > for > > > > > > > > > the > > > > > > > > > >> > >> >> moment. I > > > > > > > > > >> > >> >> > > > think > > > > > > > > > >> > >> >> > > > > it > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > difficult to keep it > > > segfault > > > > > safe > > > > > > > for > > > > > > > > > >> memory > > > > > > > > > >> > >> >> > manager, > > > > > > > > > >> > >> >> > > as > > > > > > > > > >> > >> >> > > > > > long > > > > > > > > > >> > >> >> > > > > > > as > > > > > > > > > >> > >> >> > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > always > > > > > > > > > >> > >> >> > > > > > > > > > > > > de-allocate the memory > > > segment > > > > > > when > > > > > > > it > > > > > > > > > is > > > > > > > > > >> > >> released > > > > > > > > > >> > >> >> > from > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > consumers. Only if the > > > memory > > > > > > > consumer > > > > > > > > > >> > continue > > > > > > > > > >> > >> >> using > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > > buffer > > > > > > > > > >> > >> >> > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > segment after releasing > > it, > > > in > > > > > > which > > > > > > > > > case > > > > > > > > > >> we > > > > > > > > > >> > do > > > > > > > > > >> > >> >> want > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > job > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > fail > > > > > > > > > >> > >> >> > > > > > > > > > so > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > detect the memory leak > > > early. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.2, I don't > > > think > > > > > this > > > > > > > is > > > > > > > > a > > > > > > > > > >> good > > > > > > > > > >> > >> idea. > > > > > > > > > >> > >> >> > Not > > > > > > > > > >> > >> >> > > > only > > > > > > > > > >> > >> >> > > > > > > > because > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > assumption (regular GC > is > > > > enough > > > > > > to > > > > > > > > > clean > > > > > > > > > >> > >> direct > > > > > > > > > >> > >> >> > > buffers) > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > always > > > > > > > > > >> > >> >> > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > true, but also it makes > > > harder > > > > > for > > > > > > > > > finding > > > > > > > > > >> > >> >> problems > > > > > > > > > >> > >> >> > in > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > >> > >> >> > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > overuse. E.g., user > > > configured > > > > > > some > > > > > > > > > direct > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> for > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > libraries. > > > > > > > > > >> > >> >> > > > > > > > > > > > > If the library actually > > use > > > > more > > > > > > > > direct > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> then > > > > > > > > > >> > >> >> > > > > > configured, > > > > > > > > > >> > >> >> > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > > cannot be cleaned by GC > > > > because > > > > > > they > > > > > > > > are > > > > > > > > > >> > still > > > > > > > > > >> > >> in > > > > > > > > > >> > >> >> > use, > > > > > > > > > >> > >> >> > > > may > > > > > > > > > >> > >> >> > > > > > lead > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > overuse > > > > > > > > > >> > >> >> > > > > > > > > > > > > of the total container > > > memory. > > > > > In > > > > > > > that > > > > > > > > > >> case, > > > > > > > > > >> > >> if it > > > > > > > > > >> > >> >> > > didn't > > > > > > > > > >> > >> >> > > > > > touch > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > default max direct > memory > > > > limit, > > > > > > we > > > > > > > > > cannot > > > > > > > > > >> > get > > > > > > > > > >> > >> a > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > OOM > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > > > > > will become super hard > to > > > > > > understand > > > > > > > > > which > > > > > > > > > >> > >> part of > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > >> > >> >> > > > > > > > > > > > > to be updated. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.1, it has > the > > > > > similar > > > > > > > > > >> problem as > > > > > > > > > >> > >> >> 1.2, if > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > > > exceeded > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > memory does not reach > the > > > max > > > > > > direct > > > > > > > > > >> memory > > > > > > > > > >> > >> limit > > > > > > > > > >> > >> >> > > > specified > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > dedicated parameter. I > > think > > > > it > > > > > is > > > > > > > > > >> slightly > > > > > > > > > >> > >> better > > > > > > > > > >> > >> >> > than > > > > > > > > > >> > >> >> > > > > 1.2, > > > > > > > > > >> > >> >> > > > > > > only > > > > > > > > > >> > >> >> > > > > > > > > > > because > > > > > > > > > >> > >> >> > > > > > > > > > > > > we can tune the > parameter. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 at > > 2:53 > > > > PM > > > > > > > > Stephan > > > > > > > > > >> Ewen > > > > > > > > > >> > < > > > > > > > > > >> > >> >> > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > About the > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > >> > >> discussion, > > > > > > > > > >> > >> >> > maybe > > > > > > > > > >> > >> >> > > > let > > > > > > > > > >> > >> >> > > > > > me > > > > > > > > > >> > >> >> > > > > > > > > > > summarize > > > > > > > > > >> > >> >> > > > > > > > > > > > > it a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > bit differently: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > We have the following > > two > > > > > > options: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (1) We let > > MemorySegments > > > be > > > > > > > > > >> de-allocated > > > > > > > > > >> > by > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > GC. > > > > > > > > > >> > >> >> > > > That > > > > > > > > > >> > >> >> > > > > > > makes > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > > > > > segfault > > > > > > > > > >> > >> >> > > > > > > > > > > > > > safe. But then we > need a > > > way > > > > > to > > > > > > > > > trigger > > > > > > > > > >> GC > > > > > > > > > >> > in > > > > > > > > > >> > >> >> case > > > > > > > > > >> > >> >> > > > > > > > de-allocation > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > re-allocation of a > bunch > > > of > > > > > > > segments > > > > > > > > > >> > happens > > > > > > > > > >> > >> >> > quickly, > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > often > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > case during batch > > > scheduling > > > > > or > > > > > > > task > > > > > > > > > >> > restart. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - The > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > >> (option > > > > > > > > > >> > >> 1.1) > > > > > > > > > >> > >> >> is > > > > > > > > > >> > >> >> > one > > > > > > > > > >> > >> >> > > > way > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > >> > >> >> > > > > > > do > > > > > > > > > >> > >> >> > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Another way could > be > > > to > > > > > > have a > > > > > > > > > >> > dedicated > > > > > > > > > >> > >> >> > > > bookkeeping > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > MemoryManager (option > > > 1.2), > > > > so > > > > > > > that > > > > > > > > > this > > > > > > > > > >> > is a > > > > > > > > > >> > >> >> > number > > > > > > > > > >> > >> >> > > > > > > > independent > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > parameter. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (2) We manually > allocate > > > and > > > > > > > > > de-allocate > > > > > > > > > >> > the > > > > > > > > > >> > >> >> memory > > > > > > > > > >> > >> >> > > for > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > MemorySegments > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (option 2). That way > we > > > need > > > > > not > > > > > > > > worry > > > > > > > > > >> > about > > > > > > > > > >> > >> >> > > triggering > > > > > > > > > >> > >> >> > > > > GC > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > some > > > > > > > > > >> > >> >> > > > > > > > > > > > > > threshold or > > bookkeeping, > > > > but > > > > > it > > > > > > > is > > > > > > > > > >> harder > > > > > > > > > >> > to > > > > > > > > > >> > >> >> > prevent > > > > > > > > > >> > >> >> > > > > > > > segfaults. > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be very careful about > > when > > > > we > > > > > > > > release > > > > > > > > > >> the > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > > > segments > > > > > > > > > >> > >> >> > > > > > > (only > > > > > > > > > >> > >> >> > > > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cleanup phase of the > > main > > > > > > thread). > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option > > 1.1, > > > we > > > > > > > > probably > > > > > > > > > >> need > > > > > > > > > >> > to > > > > > > > > > >> > >> >> set > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > to > > > > > > > > > >> > >> >> > > "off_heap_managed_memory + > > > > > > > > > >> > >> >> > > > > > > > > > > direct_memory" > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > have "direct_memory" > as > > a > > > > > > separate > > > > > > > > > >> reserved > > > > > > > > > >> > >> >> memory > > > > > > > > > >> > >> >> > > > pool. > > > > > > > > > >> > >> >> > > > > > > > Because > > > > > > > > > >> > >> >> > > > > > > > > if > > > > > > > > > >> > >> >> > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > just > > > > > > > > > >> > >> >> > > > > > > > > > > > > > set > > > > "-XX:MaxDirectMemorySize" > > > > > to > > > > > > > > > >> > >> >> > > > > "off_heap_managed_memory + > > > > > > > > > >> > >> >> > > > > > > > > > > > > jvm_overhead", > > > > > > > > > >> > >> >> > > > > > > > > > > > > > then there will be > times > > > > when > > > > > > that > > > > > > > > > >> entire > > > > > > > > > >> > >> >> memory is > > > > > > > > > >> > >> >> > > > > > allocated > > > > > > > > > >> > >> >> > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > buffers and we have > > > nothing > > > > > left > > > > > > > for > > > > > > > > > the > > > > > > > > > >> > JVM > > > > > > > > > >> > >> >> > > overhead. > > > > > > > > > >> > >> >> > > > So > > > > > > > > > >> > >> >> > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > either > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > >> > >> >> > > > > > > > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > way to compensate for > > that > > > > > > (again > > > > > > > > some > > > > > > > > > >> > safety > > > > > > > > > >> > >> >> > margin > > > > > > > > > >> > >> >> > > > > cutoff > > > > > > > > > >> > >> >> > > > > > > > > value) > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > will exceed container > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option > > 1.2, > > > we > > > > > > need > > > > > > > to > > > > > > > > > be > > > > > > > > > >> > aware > > > > > > > > > >> > >> >> that > > > > > > > > > >> > >> >> > it > > > > > > > > > >> > >> >> > > > > takes > > > > > > > > > >> > >> >> > > > > > > > > > elaborate > > > > > > > > > >> > >> >> > > > > > > > > > > > > logic > > > > > > > > > >> > >> >> > > > > > > > > > > > > > to push recycling of > > > direct > > > > > > > buffers > > > > > > > > > >> without > > > > > > > > > >> > >> >> always > > > > > > > > > >> > >> >> > > > > > > triggering a > > > > > > > > > >> > >> >> > > > > > > > > > full > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My first guess is that > > the > > > > > > options > > > > > > > > > will > > > > > > > > > >> be > > > > > > > > > >> > >> >> easiest > > > > > > > > > >> > >> >> > to > > > > > > > > > >> > >> >> > > > do > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > following > > > > > > > > > >> > >> >> > > > > > > > > > > > > > order: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.1 with a > > > > > dedicated > > > > > > > > > >> > direct_memory > > > > > > > > > >> > >> >> > > > parameter, > > > > > > > > > >> > >> >> > > > > as > > > > > > > > > >> > >> >> > > > > > > > > > discussed > > > > > > > > > >> > >> >> > > > > > > > > > > > > > above. We would need > to > > > > find a > > > > > > way > > > > > > > > to > > > > > > > > > >> set > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > > > > direct_memory > > > > > > > > > >> > >> >> > > > > > > > > > parameter > > > > > > > > > >> > >> >> > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > default. We could > start > > > with > > > > > 64 > > > > > > MB > > > > > > > > and > > > > > > > > > >> see > > > > > > > > > >> > >> how > > > > > > > > > >> > >> >> it > > > > > > > > > >> > >> >> > > goes > > > > > > > > > >> > >> >> > > > in > > > > > > > > > >> > >> >> > > > > > > > > practice. > > > > > > > > > >> > >> >> > > > > > > > > > > One > > > > > > > > > >> > >> >> > > > > > > > > > > > > > danger I see is that > > > setting > > > > > > this > > > > > > > > loo > > > > > > > > > >> low > > > > > > > > > >> > can > > > > > > > > > >> > >> >> > cause a > > > > > > > > > >> > >> >> > > > > bunch > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > additional > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GCs compared to before > > (we > > > > > need > > > > > > to > > > > > > > > > watch > > > > > > > > > >> > this > > > > > > > > > >> > >> >> > > > carefully). > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 2. It is > > > actually > > > > > > quite > > > > > > > > > >> simple > > > > > > > > > >> > to > > > > > > > > > >> > >> >> > > implement, > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > >> > >> >> > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > try > > > > > > > > > >> > >> >> > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > segfault safe we are > at > > > the > > > > > > > moment. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.2: We > would > > > not > > > > > > touch > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > >> > >> >> > > > > > > > > > > > > parameter > > > > > > > > > >> > >> >> > > > > > > > > > > > > > at all and assume that > > all > > > > the > > > > > > > > direct > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > > allocations > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Netty do are > infrequent > > > > enough > > > > > > to > > > > > > > be > > > > > > > > > >> > cleaned > > > > > > > > > >> > >> up > > > > > > > > > >> > >> >> > fast > > > > > > > > > >> > >> >> > > > > enough > > > > > > > > > >> > >> >> > > > > > > > > through > > > > > > > > > >> > >> >> > > > > > > > > > > > > regular > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GC. I am not sure if > > that > > > > is a > > > > > > > valid > > > > > > > > > >> > >> assumption, > > > > > > > > > >> > >> >> > > > though. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Best, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Stephan > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 > at > > > 2:16 > > > > > PM > > > > > > > > > Xintong > > > > > > > > > >> > Song > > > > > > > > > >> > >> < > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing > > your > > > > > > opinion > > > > > > > > > Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor of > > > > > > alternative > > > > > > > > 2. > > > > > > > > > I > > > > > > > > > >> was > > > > > > > > > >> > >> >> > wondering > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > Unsafe.allocate() > > > > for > > > > > > > > off-heap > > > > > > > > > >> > >> managed > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > > and > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But > > after > > > > > > giving > > > > > > > > it a > > > > > > > > > >> > second > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > >> > >> >> > > > I > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 using > > > direct > > > > > > > memory > > > > > > > > > for > > > > > > > > > >> > >> off-heap > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > concern, > > > I > > > > > > think > > > > > > > > what > > > > > > > > > >> > >> proposed > > > > > > > > > >> > >> >> in > > > > > > > > > >> > >> >> > > this > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > > memory > > > > and > > > > > > > > network > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), > > which > > > > > means > > > > > > > > they > > > > > > > > > >> are > > > > > > > > > >> > >> >> > practically > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > > > direct > > > > > > > memory. > > > > > > > > > The > > > > > > > > > >> > only > > > > > > > > > >> > >> >> parts > > > > > > > > > >> > >> >> > of > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory > are > > > task > > > > > > > > off-heap > > > > > > > > > >> > memory > > > > > > > > > >> > >> and > > > > > > > > > >> > >> >> > JVM > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly alternative > 2 > > > > > suggests > > > > > > > to > > > > > > > > > set > > > > > > > > > >> the > > > > > > > > > >> > >> JVM > > > > > > > > > >> > >> >> max > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > 1:48 > > > > > > PM > > > > > > > > Till > > > > > > > > > >> > >> Rohrmann > > > > > > > > > >> > >> >> < > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > > clarification > > > > > > > > > >> Xintong. I > > > > > > > > > >> > >> >> > > understand > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in > favour > > > of > > > > > > > option 2 > > > > > > > > > >> > because > > > > > > > > > >> > >> it > > > > > > > > > >> > >> >> > makes > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the > > direct > > > > > > > memory, I > > > > > > > > > >> fear > > > > > > > > > >> > >> that > > > > > > > > > >> > >> >> we > > > > > > > > > >> > >> >> > > might > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we > are > > > > > > currently > > > > > > > > in: > > > > > > > > > >> The > > > > > > > > > >> > >> user > > > > > > > > > >> > >> >> > might > > > > > > > > > >> > >> >> > > > see > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS > and > > > > does > > > > > > not > > > > > > > > know > > > > > > > > > >> why > > > > > > > > > >> > >> this > > > > > > > > > >> > >> >> is > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > > the > > > > > > process > > > > > > > > > memory > > > > > > > > > >> > size > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order to > > > > > > accommodate > > > > > > > > for > > > > > > > > > >> the > > > > > > > > > >> > >> extra > > > > > > > > > >> > >> >> > > direct > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > > > memory > > > > > > > budgets > > > > > > > > > >> which > > > > > > > > > >> > >> are > > > > > > > > > >> > >> >> not > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the overall > > > > memory > > > > > > > > > >> consumption. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > > 11:01 > > > > > > > AM > > > > > > > > > >> > Xintong > > > > > > > > > >> > >> >> Song < > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain > > this > > > > > with a > > > > > > > > > >> concrete > > > > > > > > > >> > >> >> example > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we > have > > > the > > > > > > > > following > > > > > > > > > >> > >> scenario. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > > > Memory: > > > > > 1GB > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > Memory > > > > (Task > > > > > > > > Off-Heap > > > > > > > > > >> > >> Memory + > > > > > > > > > >> > >> >> JVM > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory > (JVM > > > Heap > > > > > > > Memory, > > > > > > > > > JVM > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network Memory): > > > 800MB > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative > 2, > > > we > > > > > set > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative > 3, > > > we > > > > > set > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > direct > > > > > > memory > > > > > > > > > usage > > > > > > > > > >> of > > > > > > > > > >> > >> Task > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed > > 200MB, > > > > > then > > > > > > > > > >> > alternative 2 > > > > > > > > > >> > >> >> and > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > > > Setting > > > > > > larger > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the > other > > > > > memory > > > > > > > > pools. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > direct > > > > > > memory > > > > > > > > > usage > > > > > > > > > >> of > > > > > > > > > >> > >> Task > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > potentially > > > > > > exceed > > > > > > > > > 200MB, > > > > > > > > > >> > then > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > Alternative 2 > > > > > suffers > > > > > > > > from > > > > > > > > > >> > >> frequent > > > > > > > > > >> > >> >> OOM. > > > > > > > > > >> > >> >> > > To > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do > is > > to > > > > > > modify > > > > > > > > the > > > > > > > > > >> > >> >> configuration > > > > > > > > > >> > >> >> > > and > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > Off-Heap > > > > > Memory + > > > > > > > JVM > > > > > > > > > >> > >> Overhead). > > > > > > > > > >> > >> >> > Let's > > > > > > > > > >> > >> >> > > > say > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct Memory > > to > > > > > 250MB, > > > > > > > > this > > > > > > > > > >> will > > > > > > > > > >> > >> >> reduce > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to > 750MB, > > > > given > > > > > > the > > > > > > > > > total > > > > > > > > > >> > >> process > > > > > > > > > >> > >> >> > > memory > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > alternative > > > > 3, > > > > > > > there > > > > > > > > is > > > > > > > > > >> no > > > > > > > > > >> > >> >> chance of > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of exceeding > > the > > > > > total > > > > > > > > > process > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > limit, > > > > > > > > > >> > >> >> > > > but > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up > all > > > the > > > > > > > reserved > > > > > > > > > >> native > > > > > > > > > >> > >> >> memory > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > > > Metaspace), > > > > > > > if > > > > > > > > > the > > > > > > > > > >> > >> actual > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very > close > > to > > > > > > 200MB, > > > > > > > > user > > > > > > > > > >> > >> probably > > > > > > > > > >> > >> >> do > > > > > > > > > >> > >> >> > > not > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I > think > > > > from > > > > > > the > > > > > > > > > user's > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for alternative > 2 > > > may > > > > > lead > > > > > > > to > > > > > > > > > >> lower > > > > > > > > > >> > >> >> resource > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > > 10:28 > > > > > > > > AM > > > > > > > > > >> Till > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > >> > >> >> > > < > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you > have > > > to > > > > > help > > > > > > > me > > > > > > > > > >> > >> understand > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > > > memory > > > > > > under > > > > > > > > > >> > utilization > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative > 2: > > > set > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. Then > > > there > > > > > is > > > > > > > the > > > > > > > > > risk > > > > > > > > > >> > that > > > > > > > > > >> > >> >> this > > > > > > > > > >> > >> >> > > size > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of garbage > > > > > > collection > > > > > > > > and > > > > > > > > > >> > >> >> potentially > > > > > > > > > >> > >> >> > an > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative > 3: > > > set > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 2. > > > This > > > > > > would > > > > > > > of > > > > > > > > > >> course > > > > > > > > > >> > >> >> reduce > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > > alternative > > > > > 2 > > > > > > > now > > > > > > > > > >> result > > > > > > > > > >> > >> in an > > > > > > > > > >> > >> >> > > under > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > > > alternative > > > > > > 3? > > > > > > > > If > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct memory > > size > > > > and > > > > > > we > > > > > > > > use > > > > > > > > > >> only > > > > > > > > > >> > >> >> little, > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 3 > > > > results > > > > > in > > > > > > > > > memory > > > > > > > > > >> > under > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug > 13, > > > 2019 > > > > > at > > > > > > > 4:19 > > > > > > > > > PM > > > > > > > > > >> > Yang > > > > > > > > > >> > >> >> Wang < > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > xintong,till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native and > > > > Direct > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point is > > > > setting > > > > > a > > > > > > > very > > > > > > > > > >> large > > > > > > > > > >> > >> max > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > differentiate > > > > direct > > > > > > and > > > > > > > > > >> native > > > > > > > > > >> > >> >> memory. > > > > > > > > > >> > >> >> > If > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > memory > > > and > > > > > > > > framework > > > > > > > > > >> > direct > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in > favor > > of > > > > > > setting > > > > > > > > > >> direct > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > with > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > > Calculation > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree with > > > > > xintong. > > > > > > > For > > > > > > > > > Yarn > > > > > > > > > >> > and > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > >> > >> >> > > > need > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configurations > > > in > > > > > > client > > > > > > > > to > > > > > > > > > >> avoid > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > > > master. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong > Song < > > > > > > > > > >> > >> [hidden email] > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks for > > > > > replying, > > > > > > > > Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > > MemorySegment, I > > > > > > > > > think > > > > > > > > > >> > you > > > > > > > > > >> > >> are > > > > > > > > > >> > >> >> > > right > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in > the > > > > scope > > > > > > of > > > > > > > > this > > > > > > > > > >> > FLIP. > > > > > > > > > >> > >> >> This > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configure > > > memory > > > > > > pools > > > > > > > > for > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > > consumers > > > > > use > > > > > > > it. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > direct > > > > > > memory, I > > > > > > > > > think > > > > > > > > > >> > >> >> > alternative > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > reservation > > > > issue > > > > > > that > > > > > > > > > >> > >> alternative 2 > > > > > > > > > >> > >> >> > > does, > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using > memory > > > at > > > > > the > > > > > > > > > >> container > > > > > > > > > >> > >> level, > > > > > > > > > >> > >> >> > > which > > > > > > > > > >> > >> >> > > > is > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both "Task > > > > > Off-Heap > > > > > > > > > Memory" > > > > > > > > > >> and > > > > > > > > > >> > >> "JVM > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > alternative > > 2, > > > > > users > > > > > > > > might > > > > > > > > > >> > >> configure > > > > > > > > > >> > >> >> > them > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to > > avoid > > > > > > getting > > > > > > > a > > > > > > > > > >> direct > > > > > > > > > >> > >> OOM. > > > > > > > > > >> > >> >> For > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > OOM, > > so > > > > > they > > > > > > > may > > > > > > > > > not > > > > > > > > > >> > >> config > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > two > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > consequences > > > > > are > > > > > > > > risks > > > > > > > > > >> of > > > > > > > > > >> > >> >> overall > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong > Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, > Aug > > > 13, > > > > > 2019 > > > > > > > at > > > > > > > > > >> 9:39 AM > > > > > > > > > >> > >> Till > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks > for > > > > > > proposing > > > > > > > > > this > > > > > > > > > >> > FLIP > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in > > all I > > > > > think > > > > > > > it > > > > > > > > > >> already > > > > > > > > > >> > >> >> looks > > > > > > > > > >> > >> >> > > quite > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > question > > > about > > > > > > > > > allocating > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > strictly > > > > > necessary > > > > > > > to > > > > > > > > do > > > > > > > > > >> in > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > context > > > > > > > > > >> > >> >> > > > of > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be done > > as a > > > > > > follow > > > > > > > > up? > > > > > > > > > >> > Without > > > > > > > > > >> > >> >> > knowing > > > > > > > > > >> > >> >> > > > all > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that we > > > would > > > > > > widen > > > > > > > > the > > > > > > > > > >> scope > > > > > > > > > >> > >> of > > > > > > > > > >> > >> >> this > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to touch > > all > > > > the > > > > > > > > > existing > > > > > > > > > >> > call > > > > > > > > > >> > >> >> sites > > > > > > > > > >> > >> >> > of > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > > segments > > > > > > > (this > > > > > > > > > >> should > > > > > > > > > >> > >> >> mainly > > > > > > > > > >> > >> >> > be > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > memory > > > > > > > reservation > > > > > > > > > >> call > > > > > > > > > >> > to > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this > and I > > > > would > > > > > > > hope > > > > > > > > > that > > > > > > > > > >> > >> this is > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > only > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > streaming > > > job > > > > > > would > > > > > > > > have > > > > > > > > > >> with > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Concerning > > > the > > > > > > > second > > > > > > > > > open > > > > > > > > > >> > >> >> question > > > > > > > > > >> > >> >> > > about > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > memory > > > > > > > limit, I > > > > > > > > > >> would > > > > > > > > > >> > >> also > > > > > > > > > >> > >> >> be > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > leaving > it > > > > open > > > > > > > would > > > > > > > > be > > > > > > > > > >> > best. > > > > > > > > > >> > >> My > > > > > > > > > >> > >> >> > > concern > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in a > > > > similar > > > > > > > > > situation > > > > > > > > > >> as > > > > > > > > > >> > we > > > > > > > > > >> > >> >> are > > > > > > > > > >> > >> >> > now > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > different > > > > > > memory > > > > > > > > > pools > > > > > > > > > >> > are > > > > > > > > > >> > >> not > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > different > > > > > pool, > > > > > > > then > > > > > > > > > it > > > > > > > > > >> is > > > > > > > > > >> > >> quite > > > > > > > > > >> > >> >> > hard > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > process > to > > > get > > > > > > > killed > > > > > > > > > for > > > > > > > > > >> > using > > > > > > > > > >> > >> >> too > > > > > > > > > >> > >> >> > > much > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead to > a > > > > > similar > > > > > > > > > >> situation > > > > > > > > > >> > >> what > > > > > > > > > >> > >> >> we > > > > > > > > > >> > >> >> > > have > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > setting > a > > > sane > > > > > > > default > > > > > > > > > >> value > > > > > > > > > >> > >> for > > > > > > > > > >> > >> >> max > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option > to > > > > > increase > > > > > > > it > > > > > > > > if > > > > > > > > > >> he > > > > > > > > > >> > >> runs > > > > > > > > > >> > >> >> into > > > > > > > > > >> > >> >> > > an > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > how > > > > > > would > > > > > > > > > >> > >> alternative 2 > > > > > > > > > >> > >> >> > lead > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > alternative > > > 3 > > > > > > where > > > > > > > we > > > > > > > > > set > > > > > > > > > >> > the > > > > > > > > > >> > >> >> direct > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, > > Aug > > > 9, > > > > > > 2019 > > > > > > > at > > > > > > > > > >> 9:12 > > > > > > > > > >> > AM > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > the > > > > > > > > > feedback, > > > > > > > > > >> > >> Yang. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > your > > > > > > > > > comments: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > *Native > > > and > > > > > > Direct > > > > > > > > > >> Memory* > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > think > > > > > setting > > > > > > a > > > > > > > > very > > > > > > > > > >> > large > > > > > > > > > >> > >> max > > > > > > > > > >> > >> >> > > direct > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > > > sides. > > > > > > E.g., > > > > > > > we > > > > > > > > > do > > > > > > > > > >> not > > > > > > > > > >> > >> >> worry > > > > > > > > > >> > >> >> > > about > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > > allocate > > > > > > > managed > > > > > > > > / > > > > > > > > > >> > network > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > However, > > > > there > > > > > > are > > > > > > > > > also > > > > > > > > > >> > some > > > > > > > > > >> > >> >> down > > > > > > > > > >> > >> >> > > sides > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > One > > > > > thing I > > > > > > > can > > > > > > > > > >> think > > > > > > > > > >> > >> of is > > > > > > > > > >> > >> >> > that > > > > > > > > > >> > >> >> > > > if > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > killed > > > > due > > > > > to > > > > > > > > > >> overusing > > > > > > > > > >> > >> >> memory, > > > > > > > > > >> > >> >> > it > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > of > > the > > > > > memory > > > > > > > is > > > > > > > > > >> > overused. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > > Another > > > > > > down > > > > > > > > side > > > > > > > > > >> is > > > > > > > > > >> > >> that > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > memory > > > > > > > > > limit, > > > > > > > > > >> > >> because > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > means > > > we > > > > > kind > > > > > > > of > > > > > > > > > >> relay > > > > > > > > > >> > on > > > > > > > > > >> > >> >> heap > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > That > > > > > > > > could > > > > > > > > > >> be a > > > > > > > > > >> > >> >> problem > > > > > > > > > >> > >> >> > in > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > usage > > > but > > > > > not > > > > > > > > > enough > > > > > > > > > >> > heap > > > > > > > > > >> > >> >> > activity > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe > > you > > > > can > > > > > > > share > > > > > > > > > your > > > > > > > > > >> > >> reasons > > > > > > > > > >> > >> >> > for > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if > there > > > are > > > > > > > > anything > > > > > > > > > >> else > > > > > > > > > >> > I > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > Calculation* > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If > there > > > is > > > > > any > > > > > > > > > conflict > > > > > > > > > >> > >> between > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > specified, > > > > > > > > > I > > > > > > > > > >> > >> think we > > > > > > > > > >> > >> >> > > should > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > think > > > > doing > > > > > > > > checking > > > > > > > > > >> on > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > client > > > > > > > > > >> > >> >> > > > side > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we > > can > > > > > > > discover > > > > > > > > > the > > > > > > > > > >> > >> problem > > > > > > > > > >> > >> >> > > before > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > > always > > > a > > > > > good > > > > > > > > > thing. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But we > > can > > > > not > > > > > > > only > > > > > > > > > >> rely on > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > client > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > cluster > > > > > > > > > >> > >> TaskManagers > > > > > > > > > >> > >> >> on > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > different > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > and > > > > > > > > the > > > > > > > > > >> > client > > > > > > > > > >> > >> >> does > > > > > > > > > >> > >> >> > > see > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What > do > > > you > > > > > > think? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thank > > you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong > > > Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On > Thu, > > > Aug > > > > 8, > > > > > > > 2019 > > > > > > > > at > > > > > > > > > >> 5:09 > > > > > > > > > >> > >> PM > > > > > > > > > >> > >> >> Yang > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > > > xintong, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > > your > > > > > > > > > >> detailed > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > it > > > > > > > > will > > > > > > > > > be > > > > > > > > > >> > more > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > just > > > have > > > > > few > > > > > > > > > >> questions > > > > > > > > > >> > >> about > > > > > > > > > >> > >> >> it. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > > Native > > > > > > and > > > > > > > > > Direct > > > > > > > > > >> > >> Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We > do > > > not > > > > > > > > > >> differentiate > > > > > > > > > >> > >> user > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > included > > > > in > > > > > > task > > > > > > > > > >> off-heap > > > > > > > > > >> > >> >> memory. > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > > Memory > > > > > > > > > >> Calculation > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If > the > > > sum > > > > > of > > > > > > > and > > > > > > > > > >> > >> fine-grained > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > > larger > > > > > than > > > > > > > > total > > > > > > > > > >> > >> process > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we > > need > > > to > > > > > > check > > > > > > > > the > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > Song < > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > everyone, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > > > would > > > > > > like > > > > > > > to > > > > > > > > > >> start > > > > > > > > > >> > a > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > for > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > memory > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > design > > > > > > > > > "Memory > > > > > > > > > >> > >> >> Management > > > > > > > > > >> > >> >> > > and > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > updates > > > > > > > > from > > > > > > > > > >> > >> follow-up > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > This > > > > FLIP > > > > > > > > > addresses > > > > > > > > > >> > >> several > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > memory > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Different > > > > > > > > > >> > >> configuration > > > > > > > > > >> > >> >> > for > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Complex > > > > > > > and > > > > > > > > > >> > >> difficult > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Complicated, > > > > > > > > > >> > >> uncertain > > > > > > > > > >> > >> >> and > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > changes > > > > > > to > > > > > > > > > solve > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > problems > > > > > > > > > >> > >> >> > > > can > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Extend > > > > > > > > memory > > > > > > > > > >> > >> manager > > > > > > > > > >> > >> >> to > > > > > > > > > >> > >> >> > > also > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Modify > > > > > > > how > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > >> > >> >> > > memory > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > >> reservations > > > > > > > > > >> > >> and > > > > > > > > > >> > >> >> > pools. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Simplify > > > > > > > > > memory > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > find > > > > > > > more > > > > > > > > > >> > details > > > > > > > > > >> > >> in > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > note > > > > > > > > that > > > > > > > > > >> the > > > > > > > > > >> > >> early > > > > > > > > > >> > >> >> > > design > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > to > > > > > > > > > have > > > > > > > > > >> the > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > >> > >> >> > > > in > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > forward > > > > > > > > to > > > > > > > > > >> your > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 > at > > > 2:16 > > > > > PM > > > > > > > > > Xintong > > > > > > > > > >> > Song > > > > > > > > > >> > >> < > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing > > your > > > > > > opinion > > > > > > > > > Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor of > > > > > > alternative > > > > > > > > 2. > > > > > > > > > I > > > > > > > > > >> was > > > > > > > > > >> > >> >> > wondering > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > Unsafe.allocate() > > > > for > > > > > > > > off-heap > > > > > > > > > >> > >> managed > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > > and > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But > > after > > > > > > giving > > > > > > > > it a > > > > > > > > > >> > second > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > >> > >> >> > > > I > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 using > > > direct > > > > > > > memory > > > > > > > > > for > > > > > > > > > >> > >> off-heap > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > concern, > > > I > > > > > > think > > > > > > > > what > > > > > > > > > >> > >> proposed > > > > > > > > > >> > >> >> in > > > > > > > > > >> > >> >> > > this > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > > memory > > > > and > > > > > > > > network > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), > > which > > > > > means > > > > > > > > they > > > > > > > > > >> are > > > > > > > > > >> > >> >> > practically > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > > > direct > > > > > > > memory. > > > > > > > > > The > > > > > > > > > >> > only > > > > > > > > > >> > >> >> parts > > > > > > > > > >> > >> >> > of > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory > are > > > task > > > > > > > > off-heap > > > > > > > > > >> > memory > > > > > > > > > >> > >> and > > > > > > > > > >> > >> >> > JVM > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly alternative > 2 > > > > > suggests > > > > > > > to > > > > > > > > > set > > > > > > > > > >> the > > > > > > > > > >> > >> JVM > > > > > > > > > >> > >> >> max > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > 1:48 > > > > > > PM > > > > > > > > Till > > > > > > > > > >> > >> Rohrmann > > > > > > > > > >> > >> >> < > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > > clarification > > > > > > > > > >> Xintong. I > > > > > > > > > >> > >> >> > > understand > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in > favour > > > of > > > > > > > option 2 > > > > > > > > > >> > because > > > > > > > > > >> > >> it > > > > > > > > > >> > >> >> > makes > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the > > direct > > > > > > > memory, I > > > > > > > > > >> fear > > > > > > > > > >> > >> that > > > > > > > > > >> > >> >> we > > > > > > > > > >> > >> >> > > might > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we > are > > > > > > currently > > > > > > > > in: > > > > > > > > > >> The > > > > > > > > > >> > >> user > > > > > > > > > >> > >> >> > might > > > > > > > > > >> > >> >> > > > see > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS > and > > > > does > > > > > > not > > > > > > > > know > > > > > > > > > >> why > > > > > > > > > >> > >> this > > > > > > > > > >> > >> >> is > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > > the > > > > > > process > > > > > > > > > memory > > > > > > > > > >> > size > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order to > > > > > > accommodate > > > > > > > > for > > > > > > > > > >> the > > > > > > > > > >> > >> extra > > > > > > > > > >> > >> >> > > direct > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to decrease > > > memory > > > > > > > budgets > > > > > > > > > >> which > > > > > > > > > >> > >> are > > > > > > > > > >> > >> >> not > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the overall > > > > memory > > > > > > > > > >> consumption. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > > 11:01 > > > > > > > AM > > > > > > > > > >> > Xintong > > > > > > > > > >> > >> >> Song < > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain > > this > > > > > with a > > > > > > > > > >> concrete > > > > > > > > > >> > >> >> example > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we > have > > > the > > > > > > > > following > > > > > > > > > >> > >> scenario. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > > > Memory: > > > > > 1GB > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > Memory > > > > (Task > > > > > > > > Off-Heap > > > > > > > > > >> > >> Memory + > > > > > > > > > >> > >> >> JVM > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory > (JVM > > > Heap > > > > > > > Memory, > > > > > > > > > JVM > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network Memory): > > > 800MB > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative > 2, > > > we > > > > > set > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For alternative > 3, > > > we > > > > > set > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > direct > > > > > > memory > > > > > > > > > usage > > > > > > > > > >> of > > > > > > > > > >> > >> Task > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed > > 200MB, > > > > > then > > > > > > > > > >> > alternative 2 > > > > > > > > > >> > >> >> and > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > > > Setting > > > > > > larger > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the > other > > > > > memory > > > > > > > > pools. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > direct > > > > > > memory > > > > > > > > > usage > > > > > > > > > >> of > > > > > > > > > >> > >> Task > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > potentially > > > > > > exceed > > > > > > > > > 200MB, > > > > > > > > > >> > then > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > Alternative 2 > > > > > suffers > > > > > > > > from > > > > > > > > > >> > >> frequent > > > > > > > > > >> > >> >> OOM. > > > > > > > > > >> > >> >> > > To > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do > is > > to > > > > > > modify > > > > > > > > the > > > > > > > > > >> > >> >> configuration > > > > > > > > > >> > >> >> > > and > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > Off-Heap > > > > > Memory + > > > > > > > JVM > > > > > > > > > >> > >> Overhead). > > > > > > > > > >> > >> >> > Let's > > > > > > > > > >> > >> >> > > > say > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct Memory > > to > > > > > 250MB, > > > > > > > > this > > > > > > > > > >> will > > > > > > > > > >> > >> >> reduce > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to > 750MB, > > > > given > > > > > > the > > > > > > > > > total > > > > > > > > > >> > >> process > > > > > > > > > >> > >> >> > > memory > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > alternative > > > > 3, > > > > > > > there > > > > > > > > is > > > > > > > > > >> no > > > > > > > > > >> > >> >> chance of > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of exceeding > > the > > > > > total > > > > > > > > > process > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > limit, > > > > > > > > > >> > >> >> > > > but > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up > all > > > the > > > > > > > reserved > > > > > > > > > >> native > > > > > > > > > >> > >> >> memory > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > > > Metaspace), > > > > > > > if > > > > > > > > > the > > > > > > > > > >> > >> actual > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very > close > > to > > > > > > 200MB, > > > > > > > > user > > > > > > > > > >> > >> probably > > > > > > > > > >> > >> >> do > > > > > > > > > >> > >> >> > > not > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I > think > > > > from > > > > > > the > > > > > > > > > user's > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for alternative > 2 > > > may > > > > > lead > > > > > > > to > > > > > > > > > >> lower > > > > > > > > > >> > >> >> resource > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > > 10:28 > > > > > > > > AM > > > > > > > > > >> Till > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > >> > >> >> > > < > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you > have > > > to > > > > > help > > > > > > > me > > > > > > > > > >> > >> understand > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > > > memory > > > > > > under > > > > > > > > > >> > utilization > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative > 2: > > > set > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. Then > > > there > > > > > is > > > > > > > the > > > > > > > > > risk > > > > > > > > > >> > that > > > > > > > > > >> > >> >> this > > > > > > > > > >> > >> >> > > size > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of garbage > > > > > > collection > > > > > > > > and > > > > > > > > > >> > >> >> potentially > > > > > > > > > >> > >> >> > an > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - Alternative > 3: > > > set > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 2. > > > This > > > > > > would > > > > > > > of > > > > > > > > > >> course > > > > > > > > > >> > >> >> reduce > > > > > > > > > >> > >> >> > > the > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > > alternative > > > > > 2 > > > > > > > now > > > > > > > > > >> result > > > > > > > > > >> > >> in an > > > > > > > > > >> > >> >> > > under > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > > > alternative > > > > > > 3? > > > > > > > > If > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct memory > > size > > > > and > > > > > > we > > > > > > > > use > > > > > > > > > >> only > > > > > > > > > >> > >> >> little, > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative 3 > > > > results > > > > > in > > > > > > > > > memory > > > > > > > > > >> > under > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug > 13, > > > 2019 > > > > > at > > > > > > > 4:19 > > > > > > > > > PM > > > > > > > > > >> > Yang > > > > > > > > > >> > >> >> Wang < > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > xintong,till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native and > > > > Direct > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point is > > > > setting > > > > > a > > > > > > > very > > > > > > > > > >> large > > > > > > > > > >> > >> max > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > differentiate > > > > direct > > > > > > and > > > > > > > > > >> native > > > > > > > > > >> > >> >> memory. > > > > > > > > > >> > >> >> > If > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > memory > > > and > > > > > > > > framework > > > > > > > > > >> > direct > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in > favor > > of > > > > > > setting > > > > > > > > > >> direct > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > with > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > > Calculation > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree with > > > > > xintong. > > > > > > > For > > > > > > > > > Yarn > > > > > > > > > >> > and > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > >> > >> >> > > > need > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configurations > > > in > > > > > > client > > > > > > > > to > > > > > > > > > >> avoid > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > > > master. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong > Song < > > > > > > > > > >> > >> [hidden email] > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks for > > > > > replying, > > > > > > > > Till. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > > MemorySegment, I > > > > > > > > > think > > > > > > > > > >> > you > > > > > > > > > >> > >> are > > > > > > > > > >> > >> >> > > right > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in > the > > > > scope > > > > > > of > > > > > > > > this > > > > > > > > > >> > FLIP. > > > > > > > > > >> > >> >> This > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configure > > > memory > > > > > > pools > > > > > > > > for > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > > consumers > > > > > use > > > > > > > it. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > direct > > > > > > memory, I > > > > > > > > > think > > > > > > > > > >> > >> >> > alternative > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > reservation > > > > issue > > > > > > that > > > > > > > > > >> > >> alternative 2 > > > > > > > > > >> > >> >> > > does, > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using > memory > > > at > > > > > the > > > > > > > > > >> container > > > > > > > > > >> > >> level, > > > > > > > > > >> > >> >> > > which > > > > > > > > > >> > >> >> > > > is > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both "Task > > > > > Off-Heap > > > > > > > > > Memory" > > > > > > > > > >> and > > > > > > > > > >> > >> "JVM > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > alternative > > 2, > > > > > users > > > > > > > > might > > > > > > > > > >> > >> configure > > > > > > > > > >> > >> >> > them > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to > > avoid > > > > > > getting > > > > > > > a > > > > > > > > > >> direct > > > > > > > > > >> > >> OOM. > > > > > > > > > >> > >> >> For > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > OOM, > > so > > > > > they > > > > > > > may > > > > > > > > > not > > > > > > > > > >> > >> config > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > two > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > consequences > > > > > are > > > > > > > > risks > > > > > > > > > >> of > > > > > > > > > >> > >> >> overall > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong > Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, > Aug > > > 13, > > > > > 2019 > > > > > > > at > > > > > > > > > >> 9:39 AM > > > > > > > > > >> > >> Till > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks > for > > > > > > proposing > > > > > > > > > this > > > > > > > > > >> > FLIP > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in > > all I > > > > > think > > > > > > > it > > > > > > > > > >> already > > > > > > > > > >> > >> >> looks > > > > > > > > > >> > >> >> > > quite > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > question > > > about > > > > > > > > > allocating > > > > > > > > > >> > >> memory > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > strictly > > > > > necessary > > > > > > > to > > > > > > > > do > > > > > > > > > >> in > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > context > > > > > > > > > >> > >> >> > > > of > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be done > > as a > > > > > > follow > > > > > > > > up? > > > > > > > > > >> > Without > > > > > > > > > >> > >> >> > knowing > > > > > > > > > >> > >> >> > > > all > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that we > > > would > > > > > > widen > > > > > > > > the > > > > > > > > > >> scope > > > > > > > > > >> > >> of > > > > > > > > > >> > >> >> this > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to touch > > all > > > > the > > > > > > > > > existing > > > > > > > > > >> > call > > > > > > > > > >> > >> >> sites > > > > > > > > > >> > >> >> > of > > > > > > > > > >> > >> >> > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > > segments > > > > > > > (this > > > > > > > > > >> should > > > > > > > > > >> > >> >> mainly > > > > > > > > > >> > >> >> > be > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > memory > > > > > > > reservation > > > > > > > > > >> call > > > > > > > > > >> > to > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this > and I > > > > would > > > > > > > hope > > > > > > > > > that > > > > > > > > > >> > >> this is > > > > > > > > > >> > >> >> > the > > > > > > > > > >> > >> >> > > > only > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > streaming > > > job > > > > > > would > > > > > > > > have > > > > > > > > > >> with > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Concerning > > > the > > > > > > > second > > > > > > > > > open > > > > > > > > > >> > >> >> question > > > > > > > > > >> > >> >> > > about > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > memory > > > > > > > limit, I > > > > > > > > > >> would > > > > > > > > > >> > >> also > > > > > > > > > >> > >> >> be > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > leaving > it > > > > open > > > > > > > would > > > > > > > > be > > > > > > > > > >> > best. > > > > > > > > > >> > >> My > > > > > > > > > >> > >> >> > > concern > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in a > > > > similar > > > > > > > > > situation > > > > > > > > > >> as > > > > > > > > > >> > we > > > > > > > > > >> > >> >> are > > > > > > > > > >> > >> >> > now > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > different > > > > > > memory > > > > > > > > > pools > > > > > > > > > >> > are > > > > > > > > > >> > >> not > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > different > > > > > pool, > > > > > > > then > > > > > > > > > it > > > > > > > > > >> is > > > > > > > > > >> > >> quite > > > > > > > > > >> > >> >> > hard > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > process > to > > > get > > > > > > > killed > > > > > > > > > for > > > > > > > > > >> > using > > > > > > > > > >> > >> >> too > > > > > > > > > >> > >> >> > > much > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead to > a > > > > > similar > > > > > > > > > >> situation > > > > > > > > > >> > >> what > > > > > > > > > >> > >> >> we > > > > > > > > > >> > >> >> > > have > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > setting > a > > > sane > > > > > > > default > > > > > > > > > >> value > > > > > > > > > >> > >> for > > > > > > > > > >> > >> >> max > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option > to > > > > > increase > > > > > > > it > > > > > > > > if > > > > > > > > > >> he > > > > > > > > > >> > >> runs > > > > > > > > > >> > >> >> into > > > > > > > > > >> > >> >> > > an > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > how > > > > > > would > > > > > > > > > >> > >> alternative 2 > > > > > > > > > >> > >> >> > lead > > > > > > > > > >> > >> >> > > to > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > alternative > > > 3 > > > > > > where > > > > > > > we > > > > > > > > > set > > > > > > > > > >> > the > > > > > > > > > >> > >> >> direct > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On Fri, > > Aug > > > 9, > > > > > > 2019 > > > > > > > at > > > > > > > > > >> 9:12 > > > > > > > > > >> > AM > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > the > > > > > > > > > feedback, > > > > > > > > > >> > >> Yang. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > your > > > > > > > > > comments: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > *Native > > > and > > > > > > Direct > > > > > > > > > >> Memory* > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > think > > > > > setting > > > > > > a > > > > > > > > very > > > > > > > > > >> > large > > > > > > > > > >> > >> max > > > > > > > > > >> > >> >> > > direct > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > > > sides. > > > > > > E.g., > > > > > > > we > > > > > > > > > do > > > > > > > > > >> not > > > > > > > > > >> > >> >> worry > > > > > > > > > >> > >> >> > > about > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > > allocate > > > > > > > managed > > > > > > > > / > > > > > > > > > >> > network > > > > > > > > > >> > >> >> > memory > > > > > > > > > >> > >> >> > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > However, > > > > there > > > > > > are > > > > > > > > > also > > > > > > > > > >> > some > > > > > > > > > >> > >> >> down > > > > > > > > > >> > >> >> > > sides > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > One > > > > > thing I > > > > > > > can > > > > > > > > > >> think > > > > > > > > > >> > >> of is > > > > > > > > > >> > >> >> > that > > > > > > > > > >> > >> >> > > > if > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > killed > > > > due > > > > > to > > > > > > > > > >> overusing > > > > > > > > > >> > >> >> memory, > > > > > > > > > >> > >> >> > it > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > of > > the > > > > > memory > > > > > > > is > > > > > > > > > >> > overused. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > > Another > > > > > > down > > > > > > > > side > > > > > > > > > >> is > > > > > > > > > >> > >> that > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > memory > > > > > > > > > limit, > > > > > > > > > >> > >> because > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > means > > > we > > > > > kind > > > > > > > of > > > > > > > > > >> relay > > > > > > > > > >> > on > > > > > > > > > >> > >> >> heap > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > That > > > > > > > > could > > > > > > > > > >> be a > > > > > > > > > >> > >> >> problem > > > > > > > > > >> > >> >> > in > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > usage > > > but > > > > > not > > > > > > > > > enough > > > > > > > > > >> > heap > > > > > > > > > >> > >> >> > activity > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Maybe > > you > > > > can > > > > > > > share > > > > > > > > > your > > > > > > > > > >> > >> reasons > > > > > > > > > >> > >> >> > for > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if > there > > > are > > > > > > > > anything > > > > > > > > > >> else > > > > > > > > > >> > I > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > Calculation* > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If > there > > > is > > > > > any > > > > > > > > > conflict > > > > > > > > > >> > >> between > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > specified, > > > > > > > > > I > > > > > > > > > >> > >> think we > > > > > > > > > >> > >> >> > > should > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > think > > > > doing > > > > > > > > checking > > > > > > > > > >> on > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > client > > > > > > > > > >> > >> >> > > > side > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s we > > can > > > > > > > discover > > > > > > > > > the > > > > > > > > > >> > >> problem > > > > > > > > > >> > >> >> > > before > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > > always > > > a > > > > > good > > > > > > > > > thing. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But we > > can > > > > not > > > > > > > only > > > > > > > > > >> rely on > > > > > > > > > >> > >> the > > > > > > > > > >> > >> >> > > client > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > cluster > > > > > > > > > >> > >> TaskManagers > > > > > > > > > >> > >> >> on > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > different > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > and > > > > > > > > the > > > > > > > > > >> > client > > > > > > > > > >> > >> >> does > > > > > > > > > >> > >> >> > > see > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What > do > > > you > > > > > > think? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thank > > you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Xintong > > > Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On > Thu, > > > Aug > > > > 8, > > > > > > > 2019 > > > > > > > > at > > > > > > > > > >> 5:09 > > > > > > > > > >> > >> PM > > > > > > > > > >> > >> >> Yang > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > > > xintong, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > > your > > > > > > > > > >> detailed > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > it > > > > > > > > will > > > > > > > > > be > > > > > > > > > >> > more > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > >> > >> >> > > > to > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > just > > > have > > > > > few > > > > > > > > > >> questions > > > > > > > > > >> > >> about > > > > > > > > > >> > >> >> it. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > > Native > > > > > > and > > > > > > > > > Direct > > > > > > > > > >> > >> Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We > do > > > not > > > > > > > > > >> differentiate > > > > > > > > > >> > >> user > > > > > > > > > >> > >> >> > direct > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > included > > > > in > > > > > > task > > > > > > > > > >> off-heap > > > > > > > > > >> > >> >> memory. > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > > Memory > > > > > > > > > >> Calculation > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If > the > > > sum > > > > > of > > > > > > > and > > > > > > > > > >> > >> fine-grained > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > > larger > > > > > than > > > > > > > > total > > > > > > > > > >> > >> process > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we > > need > > > to > > > > > > check > > > > > > > > the > > > > > > > > > >> > memory > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > Song < > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > everyone, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > > > would > > > > > > like > > > > > > > to > > > > > > > > > >> start > > > > > > > > > >> > a > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > for > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > memory > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > design > > > > > > > > > "Memory > > > > > > > > > >> > >> >> Management > > > > > > > > > >> > >> >> > > and > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > updates > > > > > > > > from > > > > > > > > > >> > >> follow-up > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > This > > > > FLIP > > > > > > > > > addresses > > > > > > > > > >> > >> several > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > memory > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Different > > > > > > > > > >> > >> configuration > > > > > > > > > >> > >> >> > for > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Complex > > > > > > > and > > > > > > > > > >> > >> difficult > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Complicated, > > > > > > > > > >> > >> uncertain > > > > > > > > > >> > >> >> and > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > changes > > > > > > to > > > > > > > > > solve > > > > > > > > > >> > the > > > > > > > > > >> > >> >> > problems > > > > > > > > > >> > >> >> > > > can > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Extend > > > > > > > > memory > > > > > > > > > >> > >> manager > > > > > > > > > >> > >> >> to > > > > > > > > > >> > >> >> > > also > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Modify > > > > > > > how > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > >> > >> >> > > memory > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > >> reservations > > > > > > > > > >> > >> and > > > > > > > > > >> > >> >> > pools. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Simplify > > > > > > > > > memory > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > find > > > > > > > more > > > > > > > > > >> > details > > > > > > > > > >> > >> in > > > > > > > > > >> > >> >> the > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > note > > > > > > > > that > > > > > > > > > >> the > > > > > > > > > >> > >> early > > > > > > > > > >> > >> >> > > design > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > to > > > > > > > > > have > > > > > > > > > >> the > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > >> > >> >> > > > in > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > forward > > > > > > > > to > > > > > > > > > >> your > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > you~ > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > Song > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
|
|
Yes this could be the case. But from the point of the slot, it doesn't
really matter to it because it is abstracted away and can be added at a later time when really needed. Cheers, Till On Mon, Sep 16, 2019 at 11:42 AM Xintong Song <[hidden email]> wrote: > @Till > > Just trying to understand. Are you suggesting that we have both a task > executor level memory manager, and multiple slot level memory managers with > in the task executor level one? (Or what ever the components are called) > > If that is what you mean, then it makes a lot sense to me. > > Thank you~ > > Xintong Song > > > > On Mon, Sep 16, 2019 at 5:30 PM Till Rohrmann <[hidden email]> > wrote: > > > I think conceptually it makes sense to have a MemoryManager per slot. As > > Andrey said, it should not add any noticeable overhead. Moreover, I think > > this model does not contradict any of the mentioned future improvements. > > For example, a per slot MemoryManager could still be implemented such > that > > the TaskExecutor's MemoryManager keeps a reference to it and can request > to > > give up/assign more memory segments in case of revocable memory. The > > benefit I see is a better abstraction and hence a more targeted component > > which will be easier to maintain and evolve. > > > > Cheers, > > Till > > > > On Mon, Sep 16, 2019 at 11:23 AM Xintong Song <[hidden email]> > > wrote: > > > > > I would say both ways (one unified memory manager per task executor, or > > one > > > isolated memory manager per slot) should work well at the moment. I've > > not > > > yet think this through very detailedly, but just from what I can think > of > > > at the moment, I'm in favor of having one memory manager per task > > executor, > > > for the following reasons: > > > > > > - We need the memory manager to understand quota of each operator > anyway > > > (with the provided fractions), for limiting the memory usage of > > > un-revocable operators. It shouldn't be much more complicated for the > > > memory manager to isolate memory across slots. Actually I'm wondering > > > wether it requires more efforts at all compare to having one memory > > manager > > > per slot, where we also need to handle the creation and destroying of > > > memory managers. > > > > > > - It is true that we have no plan for segments caching at the moment. > > But I > > > would say it is a good opportunity for future improvement. E.g., we may > > > have some "lazy de-allocation" strategy that only de-allocate segments > if > > > the remaining available memory quota cannot satisfy new coming > > reservation > > > requests. I understand that we should not complicate the current design > > for > > > any "potential future features". My concern is that we make efforts > > > changing from one memory manager per task executor to one per slot, > with > > > the benefit (easier to isolate memory across slots) that is uncertain > to > > > me, and later discover that we may want to change it back for further > > > optimizations. > > > > > > Anyway, I do not have strong objection against having one memory > manager > > > per slot. Just raising my concerns. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Mon, Sep 16, 2019 at 4:16 PM Andrey Zagrebin <[hidden email]> > > > wrote: > > > > > > > @Xintong > > > > > > > > Just to understand, what kind of overhead do you mean by creating the > > > > memory manager for every dynamic slot? > > > > We agreed not to preallocate segments anymore and there was no plan > to > > do > > > > any segment caching at the moment, AFAIK. > > > > This means the segments are just garbage collected upon release and > not > > > > reused. > > > > If we decide to go with caching or revoking later and share pools > among > > > > slots, it would already mean mixed segment/reservation types of > memory > > > > allocation in general (like batch and streaming). > > > > Then I think memory manager would need some input from tasks because > it > > > is > > > > hard for Flink to decide how much to cache and which pool to allocate > > > from > > > > eventually. > > > > As I understand, tasks do not really need this now. That is why I > think > > > > having memory manager per slot will give more security by isolating > > slots > > > > with relatively small effort. > > > > If there is no preallocation/caching then creating the memory manager > > is > > > > almost no-op, later we can reconsider it if needed. What do you > think? > > > > > > > > Thanks, > > > > Andrey > > > > > > > > On Sat, Sep 14, 2019 at 8:11 AM Xintong Song <[hidden email]> > > > > wrote: > > > > > > > > > @Andrey, > > > > > > > > > > If we speak in the condition of dynamic slot allocation, then > DataSet > > > > tasks > > > > > should always request slots with unknown resource profiles. Then > the > > > > > allocated slots should have 1/n on-heap managed memory and 1/n > > off-heap > > > > > managed memory of the task executor, where n is the configured > > > > > 'numberOfSlot'. Given that DataSet operators only allocate segments > > (no > > > > > reservation), and does not care whether the segments are on-heap or > > > > > off-heap, I think it makes sense to wrapping both pools for DataSet > > > > > operators, with respect to the on-heap/off-heap quota of the > > allocated > > > > > slot. > > > > > > > > > > I'm not sure whether we want to create one memory manager per > slot. I > > > > think > > > > > we can have limit the quota of each slot with one memory manager > per > > > task > > > > > executor. Given the dynamic slot allocation that dynamically create > > and > > > > > destroy slots, I think having one memory manager per task executor > > may > > > > save > > > > > the overhead for frequently creating and destroying memory > managers, > > > and > > > > > provide the chance for reusing memory segments across slots. > > > > > > > > > > Thank you~ > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 11:47 PM Andrey Zagrebin < > > [hidden email] > > > > > > > > > wrote: > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > True, there would be no regression if only one type of memory is > > > > > > configured. This can be a problem only for the old jobs running > in > > a > > > > > newly > > > > > > configured cluster. > > > > > > > > > > > > About the pool type precedence, in general, it should not matter > > for > > > > the > > > > > > users which type the segments have. > > > > > > The first implementation can be just to pull from any pool, e.g. > > > empty > > > > > one > > > > > > pool firstly and then another or some other random pulling. > > > > > > This might be a problem if we mix segment allocations and > > reservation > > > > of > > > > > > memory chunks from the same memory manager. > > > > > > The reservation will be usually for a certain type of memory then > > the > > > > > task > > > > > > will probably have to also decide from which pool to allocate the > > > > > segments. > > > > > > I would suggest we create a memory manager per slot and give it > > > memory > > > > > > limit of the slot then we do not have this kind of mixed > operation > > > > > > because Dataset/Batch jobs need only segment memory allocations > and > > > > > > streaming jobs need only memory chunks for state backends as I > > > > understand > > > > > > the current plan. > > > > > > I would suggest we will look at it if we have the mixed > operations > > at > > > > > some > > > > > > point and it becomes a problem. > > > > > > > > > > > > Thanks, > > > > > > Andrey > > > > > > > > > > > > On Fri, Sep 13, 2019 at 5:24 PM Andrey Zagrebin < > > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > > From: Xintong Song <[hidden email]> > > > > > > > Date: Thu, Sep 12, 2019 at 4:21 AM > > > > > > > Subject: Re: [DISCUSS] FLIP-49: Unified Memory Configuration > for > > > > > > > TaskExecutors > > > > > > > To: dev <[hidden email]> > > > > > > > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > > > > > Thanks for bringing this up. > > > > > > > > > > > > > > If I understand correctly, this issue only occurs where the > > cluster > > > > is > > > > > > > configured with both on-heap and off-heap memory. There should > be > > > no > > > > > > > regression for clusters configured in the old way (either all > > > on-heap > > > > > or > > > > > > > all off-heap). > > > > > > > > > > > > > > I also agree that it would be good if the DataSet API jobs can > > use > > > > both > > > > > > > memory types. The only question I can see is that, from which > > pool > > > > > (heap > > > > > > / > > > > > > > off-heap) should we allocate memory for DataSet API operators? > Do > > > we > > > > > > > always prioritize one pool over the other? Or do we always > > > prioritize > > > > > the > > > > > > > pool with more available memory left? > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 8:15 PM Andrey Zagrebin < > > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > While looking more into the implementation details of Step 4, > > we > > > > > > released > > > > > > > > during some offline discussions with @Till > > > > > > > > that there can be a performance degradation for the batch > > DataSet > > > > API > > > > > > if > > > > > > > we > > > > > > > > simply continue to pull memory from the pool > > > > > > > > according the legacy option taskmanager.memory.off-heap. > > > > > > > > > > > > > > > > The reason is that if the cluster is newly configured to > > > statically > > > > > > split > > > > > > > > heap/off-heap (not like previously either heap or 0ff-heap) > > > > > > > > then the batch DataSet API jobs will be able to use only one > > type > > > > of > > > > > > > > memory. Although it does not really matter where the memory > > > > segments > > > > > > come > > > > > > > > from > > > > > > > > and potentially batch jobs can use both. Also, currently the > > > > Dataset > > > > > > API > > > > > > > > does not result in absolute resource requirements and its > batch > > > > jobs > > > > > > will > > > > > > > > always get a default share of TM resources. > > > > > > > > > > > > > > > > The suggestion is that we let the batch tasks of Dataset API > > pull > > > > > from > > > > > > > both > > > > > > > > pools according to their fair slot share of each memory type. > > > > > > > > For that we can have a special wrapping view of both pools > > which > > > > will > > > > > > > pull > > > > > > > > segments (can be randomly) according to the slot limits. > > > > > > > > The view can wrap TM level memory pools and be given to the > > Task. > > > > > > > > > > > > > > > > Best, > > > > > > > > Andrey > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:35 PM Xintong Song < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for your comments, Andrey. > > > > > > > > > > > > > > > > > > - Regarding Task Off-Heap Memory, I think you're right that > > the > > > > > user > > > > > > > need > > > > > > > > > to make sure that direct memory and native memory together > > used > > > > by > > > > > > the > > > > > > > > user > > > > > > > > > code (external libs) do not exceed the configured value. As > > far > > > > as > > > > > I > > > > > > > can > > > > > > > > > think of, there is nothing we can do about it. > > > > > > > > > > > > > > > > > > I addressed the rest of your comment in the wiki page [1]. > > > Please > > > > > > take > > > > > > > a > > > > > > > > > look. > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 6:13 PM Andrey Zagrebin < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > EDIT: sorry for confusion I meant > > > > > > > > > > taskmanager.memory.off-heap > > > > > > > > > > instead of > > > > > > > > > > setting taskmanager.memory.preallocate > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 11:29 AM Andrey Zagrebin < > > > > > > > [hidden email]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > @Xitong thanks a lot for driving the discussion. > > > > > > > > > > > > > > > > > > > > > > I also reviewed the FLIP and it looks quite good to me. > > > > > > > > > > > Here are some comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I wanted to discuss is the > > > > > backwards-compatibility > > > > > > > > with > > > > > > > > > > > the previous user setups. We could list which > options > > we > > > > > plan > > > > > > to > > > > > > > > > > deprecate. > > > > > > > > > > > From the first glance it looks possible to provide > the > > > > > > > > same/similar > > > > > > > > > > > behaviour for the setups relying on the deprecated > > > > options. > > > > > > E.g. > > > > > > > > > > > setting taskmanager.memory.preallocate to true could > > > > > override > > > > > > > the > > > > > > > > > > > new taskmanager.memory.managed.offheap-fraction to 1 > > > etc. > > > > At > > > > > > the > > > > > > > > > > moment the > > > > > > > > > > > FLIP just states that in some cases it may require > > > > > > > re-configuring > > > > > > > > of > > > > > > > > > > > cluster if migrated from prior versions. My > suggestion > > > is > > > > > that > > > > > > > we > > > > > > > > > try > > > > > > > > > > to > > > > > > > > > > > keep it backwards-compatible unless there is a good > > > reason > > > > > > like > > > > > > > > some > > > > > > > > > > major > > > > > > > > > > > complication for the implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also couple of smaller things: > > > > > > > > > > > > > > > > > > > > > > - I suggest we remove TaskExecutorSpecifics from the > > > FLIP > > > > > and > > > > > > > > leave > > > > > > > > > > > some general wording atm, like 'data structure to > > store' > > > > or > > > > > > > > 'utility > > > > > > > > > > > classes'. When the classes are implemented, we put > the > > > > > > concrete > > > > > > > > > class > > > > > > > > > > > names. This way we can avoid confusion and stale > > > > documents. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - As I understand, if user task uses native memory > > (not > > > > > direct > > > > > > > > > memory, > > > > > > > > > > > but e.g. unsafe.allocate or from external lib), > there > > > will > > > > > be > > > > > > no > > > > > > > > > > > explicit guard against exceeding 'task off heap > > memory'. > > > > > Then > > > > > > > user > > > > > > > > > > should > > > > > > > > > > > still explicitly make sure that her/his direct > buffer > > > > > > allocation > > > > > > > > > plus > > > > > > > > > > any > > > > > > > > > > > other memory usages does not exceed value announced > as > > > > 'task > > > > > > off > > > > > > > > > > heap'. I > > > > > > > > > > > guess there is no so much that can be done about it > > > except > > > > > > > > > mentioning > > > > > > > > > > in > > > > > > > > > > > docs, similar to controlling the heap state backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 10:07 AM Yang Wang < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> I also agree that all the configuration should be > > > calculated > > > > > out > > > > > > > of > > > > > > > > > > >> TaskManager. > > > > > > > > > > >> > > > > > > > > > > >> So a full configuration should be generated before > > > > TaskManager > > > > > > > > > started. > > > > > > > > > > >> > > > > > > > > > > >> Override the calculated configurations through -D now > > > seems > > > > > > > better. > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> Best, > > > > > > > > > > >> > > > > > > > > > > >> Yang > > > > > > > > > > >> > > > > > > > > > > >> Xintong Song <[hidden email]> 于2019年9月2日周一 > > > > 上午11:39写道： > > > > > > > > > > >> > > > > > > > > > > >> > I just updated the FLIP wiki page [1], with the > > > following > > > > > > > changes: > > > > > > > > > > >> > > > > > > > > > > > >> > - Network memory uses JVM direct memory, and is > > > > accounted > > > > > > > when > > > > > > > > > > >> setting > > > > > > > > > > >> > JVM max direct memory size parameter. > > > > > > > > > > >> > - Use dynamic configurations (`-Dkey=value`) to > > pass > > > > > > > calculated > > > > > > > > > > >> memory > > > > > > > > > > >> > configs into TaskExecutors, instead of ENV > > variables. > > > > > > > > > > >> > - Remove 'supporting memory reservation' from the > > > scope > > > > > of > > > > > > > this > > > > > > > > > > FLIP. > > > > > > > > > > >> > > > > > > > > > > > >> > @till @stephan, please take another look see if > there > > > are > > > > > any > > > > > > > > other > > > > > > > > > > >> > concerns. > > > > > > > > > > >> > > > > > > > > > > > >> > Thank you~ > > > > > > > > > > >> > > > > > > > > > > > >> > Xintong Song > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > [1] > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > >> > > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > >> > > > > > > > > > > > >> > > Sorry for the late response. > > > > > > > > > > >> > > > > > > > > > > > > >> > > - Regarding the `TaskExecutorSpecifics` naming, > > let's > > > > > > discuss > > > > > > > > the > > > > > > > > > > >> detail > > > > > > > > > > >> > > in PR. > > > > > > > > > > >> > > - Regarding passing parameters into the > > > `TaskExecutor`, > > > > +1 > > > > > > for > > > > > > > > > using > > > > > > > > > > >> > > dynamic configuration at the moment, given that > > there > > > > are > > > > > > more > > > > > > > > > > >> questions > > > > > > > > > > >> > to > > > > > > > > > > >> > > be discussed to have a general framework for > > > overwriting > > > > > > > > > > >> configurations > > > > > > > > > > >> > > with ENV variables. > > > > > > > > > > >> > > - Regarding memory reservation, I double checked > > with > > > Yu > > > > > and > > > > > > > he > > > > > > > > > will > > > > > > > > > > >> take > > > > > > > > > > >> > > care of it. > > > > > > > > > > >> > > > > > > > > > > > > >> > > Thank you~ > > > > > > > > > > >> > > > > > > > > > > > > >> > > Xintong Song > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann < > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > >> What I forgot to add is that we could tackle > > > specifying > > > > > the > > > > > > > > > > >> > configuration > > > > > > > > > > >> > >> fully in an incremental way and that the full > > > > > specification > > > > > > > > > should > > > > > > > > > > be > > > > > > > > > > >> > the > > > > > > > > > > >> > >> desired end state. > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann < > > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> wrote: > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > I think our goal should be that the > configuration > > > is > > > > > > fully > > > > > > > > > > >> specified > > > > > > > > > > >> > >> when > > > > > > > > > > >> > >> > the process is started. By considering the > > internal > > > > > > > > calculation > > > > > > > > > > >> step > > > > > > > > > > >> > to > > > > > > > > > > >> > >> be > > > > > > > > > > >> > >> > rather validate existing values and calculate > > > missing > > > > > > ones, > > > > > > > > > these > > > > > > > > > > >> two > > > > > > > > > > >> > >> > proposal shouldn't even conflict (given > > > determinism). > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > Since we don't want to change an existing > > > > > > flink-conf.yaml, > > > > > > > > > > >> specifying > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> > full configuration would require to pass in the > > > > options > > > > > > > > > > >> differently. > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > One way could be the ENV variables approach. > The > > > > reason > > > > > > why > > > > > > > > I'm > > > > > > > > > > >> trying > > > > > > > > > > >> > >> to > > > > > > > > > > >> > >> > exclude this feature from the FLIP is that I > > > believe > > > > it > > > > > > > > needs a > > > > > > > > > > bit > > > > > > > > > > >> > more > > > > > > > > > > >> > >> > discussion. Just some questions which come to > my > > > > mind: > > > > > > What > > > > > > > > > would > > > > > > > > > > >> be > > > > > > > > > > >> > the > > > > > > > > > > >> > >> > exact format (FLINK_KEY_NAME)? Would we > support a > > > dot > > > > > > > > separator > > > > > > > > > > >> which > > > > > > > > > > >> > is > > > > > > > > > > >> > >> > supported by some systems (FLINK.KEY.NAME)? If > > we > > > > > accept > > > > > > > the > > > > > > > > > dot > > > > > > > > > > >> > >> > separator what would be the order of precedence > > if > > > > > there > > > > > > > are > > > > > > > > > two > > > > > > > > > > >> ENV > > > > > > > > > > >> > >> > variables defined (FLINK_KEY_NAME and > > > FLINK.KEY.NAME > > > > )? > > > > > > > What > > > > > > > > is > > > > > > > > > > the > > > > > > > > > > >> > >> > precedence of env variable vs. dynamic > > > configuration > > > > > > value > > > > > > > > > > >> specified > > > > > > > > > > >> > >> via -D? > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > Another approach could be to pass in the > dynamic > > > > > > > > configuration > > > > > > > > > > >> values > > > > > > > > > > >> > >> via > > > > > > > > > > >> > >> > `-Dkey=value` to the Flink process. For that we > > > don't > > > > > > have > > > > > > > to > > > > > > > > > > >> change > > > > > > > > > > >> > >> > anything because the functionality already > > exists. > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > Cheers, > > > > > > > > > > >> > >> > Till > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan Ewen < > > > > > > > > > [hidden email]> > > > > > > > > > > >> > wrote: > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> >> I see. Under the assumption of strict > > determinism > > > > that > > > > > > > > should > > > > > > > > > > >> work. > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> The original proposal had this point "don't > > > compute > > > > > > inside > > > > > > > > the > > > > > > > > > > TM, > > > > > > > > > > >> > >> compute > > > > > > > > > > >> > >> >> outside and supply a full config", because > that > > > > > sounded > > > > > > > more > > > > > > > > > > >> > intuitive. > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till > Rohrmann < > > > > > > > > > > >> [hidden email] > > > > > > > > > > >> > > > > > > > > > > > > >> > >> >> wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> >> > My understanding was that before starting > the > > > > Flink > > > > > > > > process > > > > > > > > > we > > > > > > > > > > >> > call a > > > > > > > > > > >> > >> >> > utility which calculates these values. I > > assume > > > > that > > > > > > > this > > > > > > > > > > >> utility > > > > > > > > > > >> > >> will > > > > > > > > > > >> > >> >> do > > > > > > > > > > >> > >> >> > the calculation based on a set of configured > > > > values > > > > > > > > (process > > > > > > > > > > >> > memory, > > > > > > > > > > >> > >> >> flink > > > > > > > > > > >> > >> >> > memory, network memory etc.). Assuming that > > > these > > > > > > values > > > > > > > > > don't > > > > > > > > > > >> > differ > > > > > > > > > > >> > >> >> from > > > > > > > > > > >> > >> >> > the values with which the JVM is started, it > > > > should > > > > > be > > > > > > > > > > possible > > > > > > > > > > >> to > > > > > > > > > > >> > >> >> > recompute them in the Flink process in order > > to > > > > set > > > > > > the > > > > > > > > > > values. > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan > Ewen > > < > > > > > > > > > > [hidden email] > > > > > > > > > > >> > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > When computing the values in the JVM > process > > > > after > > > > > > it > > > > > > > > > > started, > > > > > > > > > > >> > how > > > > > > > > > > >> > >> >> would > > > > > > > > > > >> > >> >> > > you deal with values like Max Direct > Memory, > > > > > > Metaspace > > > > > > > > > size. > > > > > > > > > > >> > native > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > reservation (reduce heap size), etc? All > the > > > > > values > > > > > > > that > > > > > > > > > are > > > > > > > > > > >> > >> >> parameters > > > > > > > > > > >> > >> >> > to > > > > > > > > > > >> > >> >> > > the JVM process and that need to be > supplied > > > at > > > > > > > process > > > > > > > > > > >> startup? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till > > Rohrmann > > > < > > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > Thanks for the clarification. I have > some > > > more > > > > > > > > comments: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > - I would actually split the logic to > > > compute > > > > > the > > > > > > > > > process > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > > requirements and storing the values into > > two > > > > > > things. > > > > > > > > > E.g. > > > > > > > > > > >> one > > > > > > > > > > >> > >> could > > > > > > > > > > >> > >> >> > name > > > > > > > > > > >> > >> >> > > > the former TaskExecutorProcessUtility > and > > > the > > > > > > > latter > > > > > > > > > > >> > >> >> > > > TaskExecutorProcessMemory. But we can > > > discuss > > > > > this > > > > > > > on > > > > > > > > > the > > > > > > > > > > PR > > > > > > > > > > >> > >> since > > > > > > > > > > >> > >> >> it's > > > > > > > > > > >> > >> >> > > > just a naming detail. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > - Generally, I'm not opposed to making > > > > > > configuration > > > > > > > > > > values > > > > > > > > > > >> > >> >> overridable > > > > > > > > > > >> > >> >> > > by > > > > > > > > > > >> > >> >> > > > ENV variables. I think this is a very > good > > > > idea > > > > > > and > > > > > > > > > makes > > > > > > > > > > >> the > > > > > > > > > > >> > >> >> > > > configurability of Flink processes > easier. > > > > > > However, > > > > > > > I > > > > > > > > > > think > > > > > > > > > > >> > that > > > > > > > > > > >> > >> >> adding > > > > > > > > > > >> > >> >> > > > this functionality should not be part of > > > this > > > > > FLIP > > > > > > > > > because > > > > > > > > > > >> it > > > > > > > > > > >> > >> would > > > > > > > > > > >> > >> >> > > simply > > > > > > > > > > >> > >> >> > > > widen the scope unnecessarily. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > The reasons why I believe it is > > unnecessary > > > > are > > > > > > the > > > > > > > > > > >> following: > > > > > > > > > > >> > >> For > > > > > > > > > > >> > >> >> Yarn > > > > > > > > > > >> > >> >> > > we > > > > > > > > > > >> > >> >> > > > already create write a flink-conf.yaml > > which > > > > > could > > > > > > > be > > > > > > > > > > >> populated > > > > > > > > > > >> > >> with > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > memory settings. For the other processes > > it > > > > > should > > > > > > > not > > > > > > > > > > make > > > > > > > > > > >> a > > > > > > > > > > >> > >> >> > difference > > > > > > > > > > >> > >> >> > > > whether the loaded Configuration is > > > populated > > > > > with > > > > > > > the > > > > > > > > > > >> memory > > > > > > > > > > >> > >> >> settings > > > > > > > > > > >> > >> >> > > from > > > > > > > > > > >> > >> >> > > > ENV variables or by using > > > > > > TaskExecutorProcessUtility > > > > > > > > to > > > > > > > > > > >> compute > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > missing > > > > > > > > > > >> > >> >> > > > values from the loaded configuration. If > > the > > > > > > latter > > > > > > > > > would > > > > > > > > > > >> not > > > > > > > > > > >> > be > > > > > > > > > > >> > >> >> > possible > > > > > > > > > > >> > >> >> > > > (wrong or missing configuration values), > > > then > > > > we > > > > > > > > should > > > > > > > > > > not > > > > > > > > > > >> > have > > > > > > > > > > >> > >> >> been > > > > > > > > > > >> > >> >> > > able > > > > > > > > > > >> > >> >> > > > to actually start the process in the > first > > > > > place. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > - Concerning the memory reservation: I > > agree > > > > > with > > > > > > > you > > > > > > > > > that > > > > > > > > > > >> we > > > > > > > > > > >> > >> need > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > memory reservation functionality to make > > > > > streaming > > > > > > > > jobs > > > > > > > > > > work > > > > > > > > > > >> > with > > > > > > > > > > >> > >> >> > > "managed" > > > > > > > > > > >> > >> >> > > > memory. However, w/o this functionality > > the > > > > > whole > > > > > > > Flip > > > > > > > > > > would > > > > > > > > > > >> > >> already > > > > > > > > > > >> > >> >> > > bring > > > > > > > > > > >> > >> >> > > > a good amount of improvements to our > users > > > > when > > > > > > > > running > > > > > > > > > > >> batch > > > > > > > > > > >> > >> jobs. > > > > > > > > > > >> > >> >> > > > Moreover, by keeping the scope smaller > we > > > can > > > > > > > complete > > > > > > > > > the > > > > > > > > > > >> FLIP > > > > > > > > > > >> > >> >> faster. > > > > > > > > > > >> > >> >> > > > Hence, I would propose to address the > > memory > > > > > > > > reservation > > > > > > > > > > >> > >> >> functionality > > > > > > > > > > >> > >> >> > > as a > > > > > > > > > > >> > >> >> > > > follow up FLIP (which Yu is working on > if > > > I'm > > > > > not > > > > > > > > > > mistaken). > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > Cheers, > > > > > > > > > > >> > >> >> > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang > > Wang < > > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > Just add my 2 cents. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > Using environment variables to > override > > > the > > > > > > > > > > configuration > > > > > > > > > > >> for > > > > > > > > > > >> > >> >> > different > > > > > > > > > > >> > >> >> > > > > taskmanagers is better. > > > > > > > > > > >> > >> >> > > > > We do not need to generate dedicated > > > > > > > flink-conf.yaml > > > > > > > > > for > > > > > > > > > > >> all > > > > > > > > > > >> > >> >> > > > taskmanagers. > > > > > > > > > > >> > >> >> > > > > A common flink-conf.yam and different > > > > > > environment > > > > > > > > > > >> variables > > > > > > > > > > >> > are > > > > > > > > > > >> > >> >> > enough. > > > > > > > > > > >> > >> >> > > > > By reducing the distributed cached > > files, > > > it > > > > > > could > > > > > > > > > make > > > > > > > > > > >> > >> launching > > > > > > > > > > >> > >> >> a > > > > > > > > > > >> > >> >> > > > > taskmanager faster. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > Stephan gives a good suggestion that > we > > > > could > > > > > > move > > > > > > > > the > > > > > > > > > > >> logic > > > > > > > > > > >> > >> into > > > > > > > > > > >> > >> >> > > > > "GlobalConfiguration.loadConfig()" > > method. > > > > > > > > > > >> > >> >> > > > > Maybe the client could also benefit > from > > > > this. > > > > > > > > > Different > > > > > > > > > > >> > users > > > > > > > > > > >> > >> do > > > > > > > > > > >> > >> >> not > > > > > > > > > > >> > >> >> > > > have > > > > > > > > > > >> > >> >> > > > > to export FLINK_CONF_DIR to update few > > > > config > > > > > > > > options. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > Best, > > > > > > > > > > >> > >> >> > > > > Yang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > Stephan Ewen <[hidden email]> > > > > 于2019年8月28日周三 > > > > > > > > > 上午1:21写道： > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > One note on the Environment > Variables > > > and > > > > > > > > > > Configuration > > > > > > > > > > >> > >> >> discussion. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > My understanding is that passed ENV > > > > > variables > > > > > > > are > > > > > > > > > > added > > > > > > > > > > >> to > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > > > > configuration in the > > > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > > >> > >> method > > > > > > > > > > >> > >> >> (or > > > > > > > > > > >> > >> >> > > > > > similar). > > > > > > > > > > >> > >> >> > > > > > For all the code inside Flink, it > > looks > > > > like > > > > > > the > > > > > > > > > data > > > > > > > > > > >> was > > > > > > > > > > >> > in > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > config > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > start with, just that the scripts > that > > > > > compute > > > > > > > the > > > > > > > > > > >> > variables > > > > > > > > > > >> > >> can > > > > > > > > > > >> > >> >> > pass > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > values to the process without > actually > > > > > needing > > > > > > > to > > > > > > > > > > write > > > > > > > > > > >> a > > > > > > > > > > >> > >> file. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > For example the > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > > >> method > > > > > > > > > > >> > >> would > > > > > > > > > > >> > >> >> > take > > > > > > > > > > >> > >> >> > > > any > > > > > > > > > > >> > >> >> > > > > > ENV variable prefixed with "flink" > and > > > add > > > > > it > > > > > > > as a > > > > > > > > > > >> config > > > > > > > > > > >> > >> key. > > > > > > > > > > >> > >> >> > > > > > "flink_taskmanager_memory_size=2g" > > would > > > > > > become > > > > > > > > > > >> > >> >> > > > "taskmanager.memory.size: > > > > > > > > > > >> > >> >> > > > > > 2g". > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM > > Xintong > > > > > Song < > > > > > > > > > > >> > >> >> > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thanks for the comments, Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > I've also seen your comments on > the > > > wiki > > > > > > page, > > > > > > > > but > > > > > > > > > > >> let's > > > > > > > > > > >> > >> keep > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > > > > discussion here. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding > 'TaskExecutorSpecifics', > > > how > > > > > do > > > > > > > you > > > > > > > > > > think > > > > > > > > > > >> > about > > > > > > > > > > >> > >> >> > naming > > > > > > > > > > >> > >> >> > > it > > > > > > > > > > >> > >> >> > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > > > > > >> > >> >> > > > > > > - Regarding passing memory > > > > configurations > > > > > > into > > > > > > > > > task > > > > > > > > > > >> > >> executors, > > > > > > > > > > >> > >> >> > I'm > > > > > > > > > > >> > >> >> > > in > > > > > > > > > > >> > >> >> > > > > > favor > > > > > > > > > > >> > >> >> > > > > > > of do it via environment variables > > > > rather > > > > > > than > > > > > > > > > > >> > >> configurations, > > > > > > > > > > >> > >> >> > with > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > following two reasons. > > > > > > > > > > >> > >> >> > > > > > > - It is easier to keep the > memory > > > > > options > > > > > > > once > > > > > > > > > > >> > calculate > > > > > > > > > > >> > >> >> not to > > > > > > > > > > >> > >> >> > > be > > > > > > > > > > >> > >> >> > > > > > > changed with environment variables > > > > rather > > > > > > than > > > > > > > > > > >> > >> configurations. > > > > > > > > > > >> > >> >> > > > > > > - I'm not sure whether we should > > > write > > > > > the > > > > > > > > > > >> > configuration > > > > > > > > > > >> > >> in > > > > > > > > > > >> > >> >> > > startup > > > > > > > > > > >> > >> >> > > > > > > scripts. Writing changes into the > > > > > > > configuration > > > > > > > > > > files > > > > > > > > > > >> > when > > > > > > > > > > >> > >> >> > running > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > startup scripts does not sounds > > right > > > to > > > > > me. > > > > > > > Or > > > > > > > > we > > > > > > > > > > >> could > > > > > > > > > > >> > >> make > > > > > > > > > > >> > >> >> a > > > > > > > > > > >> > >> >> > > copy > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > >> > >> >> > > > > > > configuration files per flink > > cluster, > > > > and > > > > > > > make > > > > > > > > > the > > > > > > > > > > >> task > > > > > > > > > > >> > >> >> executor > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > load > > > > > > > > > > >> > >> >> > > > > > > from the copy, and clean up the > copy > > > > after > > > > > > the > > > > > > > > > > >> cluster is > > > > > > > > > > >> > >> >> > shutdown, > > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > > >> > >> >> > > > > > > is complicated. (I think this is > > also > > > > what > > > > > > > > Stephan > > > > > > > > > > >> means > > > > > > > > > > >> > in > > > > > > > > > > >> > >> >> his > > > > > > > > > > >> > >> >> > > > comment > > > > > > > > > > >> > >> >> > > > > > on > > > > > > > > > > >> > >> >> > > > > > > the wiki page?) > > > > > > > > > > >> > >> >> > > > > > > - Regarding reserving memory, I > > think > > > > this > > > > > > > > change > > > > > > > > > > >> should > > > > > > > > > > >> > be > > > > > > > > > > >> > >> >> > > included > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > >> > >> >> > > > > > > this FLIP. I think a big part of > > > > > motivations > > > > > > > of > > > > > > > > > this > > > > > > > > > > >> FLIP > > > > > > > > > > >> > >> is > > > > > > > > > > >> > >> >> to > > > > > > > > > > >> > >> >> > > unify > > > > > > > > > > >> > >> >> > > > > > > memory configuration for > streaming / > > > > batch > > > > > > and > > > > > > > > > make > > > > > > > > > > it > > > > > > > > > > >> > easy > > > > > > > > > > >> > >> >> for > > > > > > > > > > >> > >> >> > > > > > configuring > > > > > > > > > > >> > >> >> > > > > > > rocksdb memory. If we don't > support > > > > memory > > > > > > > > > > >> reservation, > > > > > > > > > > >> > >> then > > > > > > > > > > >> > >> >> > > > streaming > > > > > > > > > > >> > >> >> > > > > > jobs > > > > > > > > > > >> > >> >> > > > > > > cannot use managed memory (neither > > > > on-heap > > > > > > or > > > > > > > > > > >> off-heap), > > > > > > > > > > >> > >> which > > > > > > > > > > >> > >> >> > > makes > > > > > > > > > > >> > >> >> > > > > this > > > > > > > > > > >> > >> >> > > > > > > FLIP incomplete. > > > > > > > > > > >> > >> >> > > > > > > - Regarding network memory, I > think > > > you > > > > > are > > > > > > > > > right. I > > > > > > > > > > >> > think > > > > > > > > > > >> > >> we > > > > > > > > > > >> > >> >> > > > probably > > > > > > > > > > >> > >> >> > > > > > > don't need to change network stack > > > from > > > > > > using > > > > > > > > > direct > > > > > > > > > > >> > >> memory to > > > > > > > > > > >> > >> >> > > using > > > > > > > > > > >> > >> >> > > > > > unsafe > > > > > > > > > > >> > >> >> > > > > > > native memory. Network memory size > > is > > > > > > > > > deterministic, > > > > > > > > > > >> > >> cannot be > > > > > > > > > > >> > >> >> > > > reserved > > > > > > > > > > >> > >> >> > > > > > as > > > > > > > > > > >> > >> >> > > > > > > managed memory does, and cannot be > > > > > > overused. I > > > > > > > > > think > > > > > > > > > > >> it > > > > > > > > > > >> > >> also > > > > > > > > > > >> > >> >> > works > > > > > > > > > > >> > >> >> > > if > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > >> > >> >> > > > > > > simply keep using direct memory > for > > > > > network > > > > > > > and > > > > > > > > > > >> include > > > > > > > > > > >> > it > > > > > > > > > > >> > >> in > > > > > > > > > > >> > >> >> jvm > > > > > > > > > > >> > >> >> > > max > > > > > > > > > > >> > >> >> > > > > > > direct memory size. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM > Till > > > > > > Rohrmann > > > > > > > < > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Hi Xintong, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > thanks for addressing the > comments > > > and > > > > > > > adding > > > > > > > > a > > > > > > > > > > more > > > > > > > > > > >> > >> >> detailed > > > > > > > > > > >> > >> >> > > > > > > > implementation plan. I have a > > couple > > > > of > > > > > > > > comments > > > > > > > > > > >> > >> concerning > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > > > > > implementation plan: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > - The name > `TaskExecutorSpecifics` > > > is > > > > > not > > > > > > > > really > > > > > > > > > > >> > >> >> descriptive. > > > > > > > > > > >> > >> >> > > > > Choosing > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > different name could help here. > > > > > > > > > > >> > >> >> > > > > > > > - I'm not sure whether I would > > pass > > > > the > > > > > > > memory > > > > > > > > > > >> > >> >> configuration to > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > > > TaskExecutor via environment > > > > variables. > > > > > I > > > > > > > > think > > > > > > > > > it > > > > > > > > > > >> > would > > > > > > > > > > >> > >> be > > > > > > > > > > >> > >> >> > > better > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > write > > > > > > > > > > >> > >> >> > > > > > > > it into the configuration one > uses > > > to > > > > > > start > > > > > > > > the > > > > > > > > > TM > > > > > > > > > > >> > >> process. > > > > > > > > > > >> > >> >> > > > > > > > - If possible, I would exclude > the > > > > > memory > > > > > > > > > > >> reservation > > > > > > > > > > >> > >> from > > > > > > > > > > >> > >> >> this > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > add this as part of a dedicated > > > FLIP. > > > > > > > > > > >> > >> >> > > > > > > > - If possible, then I would > > exclude > > > > > > changes > > > > > > > to > > > > > > > > > the > > > > > > > > > > >> > >> network > > > > > > > > > > >> > >> >> > stack > > > > > > > > > > >> > >> >> > > > from > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > FLIP. Maybe we can simply say > that > > > the > > > > > > > direct > > > > > > > > > > memory > > > > > > > > > > >> > >> needed > > > > > > > > > > >> > >> >> by > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > > network > > > > > > > > > > >> > >> >> > > > > > > > stack is the framework direct > > memory > > > > > > > > > requirement. > > > > > > > > > > >> > >> Changing > > > > > > > > > > >> > >> >> how > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > is allocated can happen in a > > second > > > > > step. > > > > > > > This > > > > > > > > > > would > > > > > > > > > > >> > keep > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > scope > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > FLIP smaller. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 PM > > > > Xintong > > > > > > > Song < > > > > > > > > > > >> > >> >> > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Hi everyone, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > I just updated the FLIP > document > > > on > > > > > wiki > > > > > > > > [1], > > > > > > > > > > with > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > > following > > > > > > > > > > >> > >> >> > > > > > > changes. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > - Removed open question > > > regarding > > > > > > > > > > MemorySegment > > > > > > > > > > >> > >> >> > allocation. > > > > > > > > > > >> > >> >> > > As > > > > > > > > > > >> > >> >> > > > > > > > > discussed, we exclude this > > > topic > > > > > from > > > > > > > the > > > > > > > > > > >> scope of > > > > > > > > > > >> > >> this > > > > > > > > > > >> > >> >> > > FLIP. > > > > > > > > > > >> > >> >> > > > > > > > > - Updated content about JVM > > > > direct > > > > > > > memory > > > > > > > > > > >> > parameter > > > > > > > > > > >> > >> >> > > according > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > recent > > > > > > > > > > >> > >> >> > > > > > > > > discussions, and moved the > > > other > > > > > > > options > > > > > > > > to > > > > > > > > > > >> > >> "Rejected > > > > > > > > > > >> > >> >> > > > > > Alternatives" > > > > > > > > > > >> > >> >> > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > moment. > > > > > > > > > > >> > >> >> > > > > > > > > - Added implementation > steps. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > [1] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 > PM > > > > > Stephan > > > > > > > > Ewen < > > > > > > > > > > >> > >> >> > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong: Concerning "wait > for > > > > > memory > > > > > > > > users > > > > > > > > > > >> before > > > > > > > > > > >> > >> task > > > > > > > > > > >> > >> >> > > dispose > > > > > > > > > > >> > >> >> > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > release": I agree, that's > how > > it > > > > > > should > > > > > > > > be. > > > > > > > > > > >> Let's > > > > > > > > > > >> > >> try it > > > > > > > > > > >> > >> >> > out. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong @Jingsong: > > Concerning " > > > > JVM > > > > > > > does > > > > > > > > > not > > > > > > > > > > >> wait > > > > > > > > > > >> > >> for > > > > > > > > > > >> > >> >> GC > > > > > > > > > > >> > >> >> > > when > > > > > > > > > > >> > >> >> > > > > > > > allocating > > > > > > > > > > >> > >> >> > > > > > > > > > direct memory buffer": There > > > seems > > > > > to > > > > > > be > > > > > > > > > > pretty > > > > > > > > > > >> > >> >> elaborate > > > > > > > > > > >> > >> >> > > logic > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > free > > > > > > > > > > >> > >> >> > > > > > > > > > buffers when allocating new > > > ones. > > > > > See > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Till: Maybe. If we assume > > that > > > > the > > > > > > JVM > > > > > > > > > > default > > > > > > > > > > >> > works > > > > > > > > > > >> > >> >> (like > > > > > > > > > > >> > >> >> > > > going > > > > > > > > > > >> > >> >> > > > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > option 2 and not setting > > > > > > > > > > >> "-XX:MaxDirectMemorySize" > > > > > > > > > > >> > at > > > > > > > > > > >> > >> >> all), > > > > > > > > > > >> > >> >> > > > then > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > >> > >> >> > > > > > > > think > > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > > should be okay to set > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > >> to > > > > > > > > > > >> > >> >> > > > > > > > > > "off_heap_managed_memory + > > > > > > > direct_memory" > > > > > > > > > even > > > > > > > > > > >> if > > > > > > > > > > >> > we > > > > > > > > > > >> > >> use > > > > > > > > > > >> > >> >> > > > RocksDB. > > > > > > > > > > >> > >> >> > > > > > > That > > > > > > > > > > >> > >> >> > > > > > > > > is a > > > > > > > > > > >> > >> >> > > > > > > > > > big if, though, I honestly > > have > > > no > > > > > > idea > > > > > > > :D > > > > > > > > > > >> Would be > > > > > > > > > > >> > >> >> good to > > > > > > > > > > >> > >> >> > > > > > > understand > > > > > > > > > > >> > >> >> > > > > > > > > > this, though, because this > > would > > > > > > affect > > > > > > > > > option > > > > > > > > > > >> (2) > > > > > > > > > > >> > >> and > > > > > > > > > > >> > >> >> > option > > > > > > > > > > >> > >> >> > > > > > (1.2). > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at 4:44 > > PM > > > > > > Xintong > > > > > > > > > Song < > > > > > > > > > > >> > >> >> > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thanks for the inputs, > > > Jingsong. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Let me try to summarize > your > > > > > points. > > > > > > > > > Please > > > > > > > > > > >> > correct > > > > > > > > > > >> > >> >> me if > > > > > > > > > > >> > >> >> > > I'm > > > > > > > > > > >> > >> >> > > > > > > wrong. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - Memory consumers > should > > > > > always > > > > > > > > avoid > > > > > > > > > > >> > returning > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > > > > segments > > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > memory manager while > > there > > > > are > > > > > > > still > > > > > > > > > > >> > un-cleaned > > > > > > > > > > >> > >> >> > > > structures / > > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > may use the memory. > > > > Otherwise, > > > > > it > > > > > > > > would > > > > > > > > > > >> cause > > > > > > > > > > >> > >> >> serious > > > > > > > > > > >> > >> >> > > > > problems > > > > > > > > > > >> > >> >> > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > >> > >> >> > > > > > > > > > > multiple consumers > trying > > > to > > > > > use > > > > > > > the > > > > > > > > > same > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > segment. > > > > > > > > > > >> > >> >> > > > > > > > > > > - JVM does not wait for > > GC > > > > when > > > > > > > > > > allocating > > > > > > > > > > >> > >> direct > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > > > > buffer. > > > > > > > > > > >> > >> >> > > > > > > > > > > Therefore even we set > > > proper > > > > > max > > > > > > > > direct > > > > > > > > > > >> memory > > > > > > > > > > >> > >> size > > > > > > > > > > >> > >> >> > > limit, > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > >> > >> >> > > > > > > may > > > > > > > > > > >> > >> >> > > > > > > > > > still > > > > > > > > > > >> > >> >> > > > > > > > > > > encounter direct memory > > oom > > > > if > > > > > > the > > > > > > > GC > > > > > > > > > > >> cleaning > > > > > > > > > > >> > >> >> memory > > > > > > > > > > >> > >> >> > > > slower > > > > > > > > > > >> > >> >> > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > direct memory > allocation. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Am I understanding this > > > > correctly? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at > 4:21 > > > PM > > > > > > > > > JingsongLee > > > > > > > > > > < > > > > > > > > > > >> > >> >> > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > .invalid> > > > > > > > > > > >> > >> >> > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Hi stephan: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About option 2: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > if additional threads > not > > > > > cleanly > > > > > > > shut > > > > > > > > > > down > > > > > > > > > > >> > >> before > > > > > > > > > > >> > >> >> we > > > > > > > > > > >> > >> >> > can > > > > > > > > > > >> > >> >> > > > > exit > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > task: > > > > > > > > > > >> > >> >> > > > > > > > > > > > In the current case of > > > memory > > > > > > reuse, > > > > > > > > it > > > > > > > > > > has > > > > > > > > > > >> > >> freed up > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > > > > uses. If this memory is > > > used > > > > by > > > > > > > other > > > > > > > > > > tasks > > > > > > > > > > >> > and > > > > > > > > > > >> > >> >> > > > asynchronous > > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > > >> > >> >> > > > > > > > > > > > of exited task may > still > > be > > > > > > > writing, > > > > > > > > > > there > > > > > > > > > > >> > will > > > > > > > > > > >> > >> be > > > > > > > > > > >> > >> >> > > > > concurrent > > > > > > > > > > >> > >> >> > > > > > > > > security > > > > > > > > > > >> > >> >> > > > > > > > > > > > problems, and even lead > > to > > > > > errors > > > > > > > in > > > > > > > > > user > > > > > > > > > > >> > >> computing > > > > > > > > > > >> > >> >> > > > results. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > So I think this is a > > serious > > > > and > > > > > > > > > > intolerable > > > > > > > > > > >> > >> bug, No > > > > > > > > > > >> > >> >> > > matter > > > > > > > > > > >> > >> >> > > > > > what > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > option is, it should be > > > > > avoided. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About direct memory > > cleaned > > > by > > > > > GC: > > > > > > > > > > >> > >> >> > > > > > > > > > > > I don't think it is a > good > > > > idea, > > > > > > > I've > > > > > > > > > > >> > >> encountered so > > > > > > > > > > >> > >> >> > many > > > > > > > > > > >> > >> >> > > > > > > > situations > > > > > > > > > > >> > >> >> > > > > > > > > > > > that it's too late for > GC > > > to > > > > > > cause > > > > > > > > > > >> > DirectMemory > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > >> > >> >> > > > Release > > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > allocate DirectMemory > > > depend > > > > on > > > > > > the > > > > > > > > > type > > > > > > > > > > of > > > > > > > > > > >> > user > > > > > > > > > > >> > >> >> job, > > > > > > > > > > >> > >> >> > > > which > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > often beyond our > control. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Best, > > > > > > > > > > >> > >> >> > > > > > > > > > > > Jingsong Lee > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > >> > >> >> > > > > > > > > > > > From:Stephan Ewen < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Send > Time:2019年8月19日(星期一) > > > > 15:56 > > > > > > > > > > >> > >> >> > > > > > > > > > > > To:dev < > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] > > > FLIP-49: > > > > > > > Unified > > > > > > > > > > >> Memory > > > > > > > > > > >> > >> >> > > Configuration > > > > > > > > > > >> > >> >> > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > TaskExecutors > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > My main concern with > > option > > > 2 > > > > > > > > (manually > > > > > > > > > > >> release > > > > > > > > > > >> > >> >> memory) > > > > > > > > > > >> > >> >> > > is > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > segfaults > > > > > > > > > > >> > >> >> > > > > > > > > > > > in the JVM send off all > > > sorts > > > > of > > > > > > > > alarms > > > > > > > > > on > > > > > > > > > > >> user > > > > > > > > > > >> > >> >> ends. > > > > > > > > > > >> > >> >> > So > > > > > > > > > > >> > >> >> > > we > > > > > > > > > > >> > >> >> > > > > > need > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > guarantee that this > never > > > > > happens. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > The trickyness is in > tasks > > > > that > > > > > > uses > > > > > > > > > data > > > > > > > > > > >> > >> >> structures / > > > > > > > > > > >> > >> >> > > > > > algorithms > > > > > > > > > > >> > >> >> > > > > > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > additional threads, like > > > hash > > > > > > table > > > > > > > > > > >> spill/read > > > > > > > > > > >> > >> and > > > > > > > > > > >> > >> >> > > sorting > > > > > > > > > > >> > >> >> > > > > > > threads. > > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > > >> > >> >> > > > > > > > > > > > to ensure that these > > cleanly > > > > > shut > > > > > > > down > > > > > > > > > > >> before > > > > > > > > > > >> > we > > > > > > > > > > >> > >> can > > > > > > > > > > >> > >> >> > exit > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > task. > > > > > > > > > > >> > >> >> > > > > > > > > > > > I am not sure that we > have > > > > that > > > > > > > > > guaranteed > > > > > > > > > > >> > >> already, > > > > > > > > > > >> > >> >> > > that's > > > > > > > > > > >> > >> >> > > > > why > > > > > > > > > > >> > >> >> > > > > > > > option > > > > > > > > > > >> > >> >> > > > > > > > > > 1.1 > > > > > > > > > > >> > >> >> > > > > > > > > > > > seemed simpler to me. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 at > > 3:42 > > > > PM > > > > > > > > Xintong > > > > > > > > > > >> Song < > > > > > > > > > > >> > >> >> > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thanks for the > comments, > > > > > > Stephan. > > > > > > > > > > >> Summarized > > > > > > > > > > >> > in > > > > > > > > > > >> > >> >> this > > > > > > > > > > >> > >> >> > > way > > > > > > > > > > >> > >> >> > > > > > really > > > > > > > > > > >> > >> >> > > > > > > > > makes > > > > > > > > > > >> > >> >> > > > > > > > > > > > > things easier to > > > understand. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'm in favor of option > > 2, > > > at > > > > > > least > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > >> > >> >> moment. I > > > > > > > > > > >> > >> >> > > > think > > > > > > > > > > >> > >> >> > > > > it > > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > difficult to keep it > > > > segfault > > > > > > safe > > > > > > > > for > > > > > > > > > > >> memory > > > > > > > > > > >> > >> >> > manager, > > > > > > > > > > >> > >> >> > > as > > > > > > > > > > >> > >> >> > > > > > long > > > > > > > > > > >> > >> >> > > > > > > as > > > > > > > > > > >> > >> >> > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > always > > > > > > > > > > >> > >> >> > > > > > > > > > > > > de-allocate the memory > > > > segment > > > > > > > when > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > >> > >> released > > > > > > > > > > >> > >> >> > from > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > consumers. Only if the > > > > memory > > > > > > > > consumer > > > > > > > > > > >> > continue > > > > > > > > > > >> > >> >> using > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > > buffer > > > > > > > > > > >> > >> >> > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > segment after > releasing > > > it, > > > > in > > > > > > > which > > > > > > > > > > case > > > > > > > > > > >> we > > > > > > > > > > >> > do > > > > > > > > > > >> > >> >> want > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > job > > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > fail > > > > > > > > > > >> > >> >> > > > > > > > > > so > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > detect the memory leak > > > > early. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.2, I > don't > > > > think > > > > > > this > > > > > > > > is > > > > > > > > > a > > > > > > > > > > >> good > > > > > > > > > > >> > >> idea. > > > > > > > > > > >> > >> >> > Not > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > >> > >> >> > > > > > > > because > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > assumption (regular GC > > is > > > > > enough > > > > > > > to > > > > > > > > > > clean > > > > > > > > > > >> > >> direct > > > > > > > > > > >> > >> >> > > buffers) > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > always > > > > > > > > > > >> > >> >> > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > true, but also it > makes > > > > harder > > > > > > for > > > > > > > > > > finding > > > > > > > > > > >> > >> >> problems > > > > > > > > > > >> > >> >> > in > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > >> > >> >> > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > overuse. E.g., user > > > > configured > > > > > > > some > > > > > > > > > > direct > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> for > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > libraries. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > If the library > actually > > > use > > > > > more > > > > > > > > > direct > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> then > > > > > > > > > > >> > >> >> > > > > > configured, > > > > > > > > > > >> > >> >> > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cannot be cleaned by > GC > > > > > because > > > > > > > they > > > > > > > > > are > > > > > > > > > > >> > still > > > > > > > > > > >> > >> in > > > > > > > > > > >> > >> >> > use, > > > > > > > > > > >> > >> >> > > > may > > > > > > > > > > >> > >> >> > > > > > lead > > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > overuse > > > > > > > > > > >> > >> >> > > > > > > > > > > > > of the total container > > > > memory. > > > > > > In > > > > > > > > that > > > > > > > > > > >> case, > > > > > > > > > > >> > >> if it > > > > > > > > > > >> > >> >> > > didn't > > > > > > > > > > >> > >> >> > > > > > touch > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > default max direct > > memory > > > > > limit, > > > > > > > we > > > > > > > > > > cannot > > > > > > > > > > >> > get > > > > > > > > > > >> > >> a > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > OOM > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > > > > > will become super hard > > to > > > > > > > understand > > > > > > > > > > which > > > > > > > > > > >> > >> part of > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to be updated. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.1, it has > > the > > > > > > similar > > > > > > > > > > >> problem as > > > > > > > > > > >> > >> >> 1.2, if > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > > > exceeded > > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > memory does not reach > > the > > > > max > > > > > > > direct > > > > > > > > > > >> memory > > > > > > > > > > >> > >> limit > > > > > > > > > > >> > >> >> > > > specified > > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > dedicated parameter. I > > > think > > > > > it > > > > > > is > > > > > > > > > > >> slightly > > > > > > > > > > >> > >> better > > > > > > > > > > >> > >> >> > than > > > > > > > > > > >> > >> >> > > > > 1.2, > > > > > > > > > > >> > >> >> > > > > > > only > > > > > > > > > > >> > >> >> > > > > > > > > > > because > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we can tune the > > parameter. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 > at > > > 2:53 > > > > > PM > > > > > > > > > Stephan > > > > > > > > > > >> Ewen > > > > > > > > > > >> > < > > > > > > > > > > >> > >> >> > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > About the > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > >> > >> discussion, > > > > > > > > > > >> > >> >> > maybe > > > > > > > > > > >> > >> >> > > > let > > > > > > > > > > >> > >> >> > > > > > me > > > > > > > > > > >> > >> >> > > > > > > > > > > summarize > > > > > > > > > > >> > >> >> > > > > > > > > > > > > it a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > bit differently: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > We have the > following > > > two > > > > > > > options: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (1) We let > > > MemorySegments > > > > be > > > > > > > > > > >> de-allocated > > > > > > > > > > >> > by > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > GC. > > > > > > > > > > >> > >> >> > > > That > > > > > > > > > > >> > >> >> > > > > > > makes > > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > > > > > segfault > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > safe. But then we > > need a > > > > way > > > > > > to > > > > > > > > > > trigger > > > > > > > > > > >> GC > > > > > > > > > > >> > in > > > > > > > > > > >> > >> >> case > > > > > > > > > > >> > >> >> > > > > > > > de-allocation > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > re-allocation of a > > bunch > > > > of > > > > > > > > segments > > > > > > > > > > >> > happens > > > > > > > > > > >> > >> >> > quickly, > > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > often > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > case during batch > > > > scheduling > > > > > > or > > > > > > > > task > > > > > > > > > > >> > restart. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - The > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > >> (option > > > > > > > > > > >> > >> 1.1) > > > > > > > > > > >> > >> >> is > > > > > > > > > > >> > >> >> > one > > > > > > > > > > >> > >> >> > > > way > > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > > >> > >> >> > > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Another way > could > > be > > > > to > > > > > > > have a > > > > > > > > > > >> > dedicated > > > > > > > > > > >> > >> >> > > > bookkeeping > > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > MemoryManager > (option > > > > 1.2), > > > > > so > > > > > > > > that > > > > > > > > > > this > > > > > > > > > > >> > is a > > > > > > > > > > >> > >> >> > number > > > > > > > > > > >> > >> >> > > > > > > > independent > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > parameter. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (2) We manually > > allocate > > > > and > > > > > > > > > > de-allocate > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> memory > > > > > > > > > > >> > >> >> > > for > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > MemorySegments > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (option 2). That way > > we > > > > need > > > > > > not > > > > > > > > > worry > > > > > > > > > > >> > about > > > > > > > > > > >> > >> >> > > triggering > > > > > > > > > > >> > >> >> > > > > GC > > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > some > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > threshold or > > > bookkeeping, > > > > > but > > > > > > it > > > > > > > > is > > > > > > > > > > >> harder > > > > > > > > > > >> > to > > > > > > > > > > >> > >> >> > prevent > > > > > > > > > > >> > >> >> > > > > > > > segfaults. > > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be very careful > about > > > when > > > > > we > > > > > > > > > release > > > > > > > > > > >> the > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > > > segments > > > > > > > > > > >> > >> >> > > > > > > (only > > > > > > > > > > >> > >> >> > > > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cleanup phase of the > > > main > > > > > > > thread). > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option > > > 1.1, > > > > we > > > > > > > > > probably > > > > > > > > > > >> need > > > > > > > > > > >> > to > > > > > > > > > > >> > >> >> set > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > to > > > > > > > > > > >> > >> >> > > "off_heap_managed_memory + > > > > > > > > > > >> > >> >> > > > > > > > > > > direct_memory" > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > have "direct_memory" > > as > > > a > > > > > > > separate > > > > > > > > > > >> reserved > > > > > > > > > > >> > >> >> memory > > > > > > > > > > >> > >> >> > > > pool. > > > > > > > > > > >> > >> >> > > > > > > > Because > > > > > > > > > > >> > >> >> > > > > > > > > if > > > > > > > > > > >> > >> >> > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > just > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > set > > > > > "-XX:MaxDirectMemorySize" > > > > > > to > > > > > > > > > > >> > >> >> > > > > "off_heap_managed_memory + > > > > > > > > > > >> > >> >> > > > > > > > > > > > > jvm_overhead", > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > then there will be > > times > > > > > when > > > > > > > that > > > > > > > > > > >> entire > > > > > > > > > > >> > >> >> memory is > > > > > > > > > > >> > >> >> > > > > > allocated > > > > > > > > > > >> > >> >> > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > buffers and we have > > > > nothing > > > > > > left > > > > > > > > for > > > > > > > > > > the > > > > > > > > > > >> > JVM > > > > > > > > > > >> > >> >> > > overhead. > > > > > > > > > > >> > >> >> > > > So > > > > > > > > > > >> > >> >> > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > either > > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > > >> > >> >> > > > > > > > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > way to compensate > for > > > that > > > > > > > (again > > > > > > > > > some > > > > > > > > > > >> > safety > > > > > > > > > > >> > >> >> > margin > > > > > > > > > > >> > >> >> > > > > cutoff > > > > > > > > > > >> > >> >> > > > > > > > > value) > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > will exceed > container > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with option > > > 1.2, > > > > we > > > > > > > need > > > > > > > > to > > > > > > > > > > be > > > > > > > > > > >> > aware > > > > > > > > > > >> > >> >> that > > > > > > > > > > >> > >> >> > it > > > > > > > > > > >> > >> >> > > > > takes > > > > > > > > > > >> > >> >> > > > > > > > > > elaborate > > > > > > > > > > >> > >> >> > > > > > > > > > > > > logic > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > to push recycling of > > > > direct > > > > > > > > buffers > > > > > > > > > > >> without > > > > > > > > > > >> > >> >> always > > > > > > > > > > >> > >> >> > > > > > > triggering a > > > > > > > > > > >> > >> >> > > > > > > > > > full > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My first guess is > that > > > the > > > > > > > options > > > > > > > > > > will > > > > > > > > > > >> be > > > > > > > > > > >> > >> >> easiest > > > > > > > > > > >> > >> >> > to > > > > > > > > > > >> > >> >> > > > do > > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > following > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > order: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.1 with > a > > > > > > dedicated > > > > > > > > > > >> > direct_memory > > > > > > > > > > >> > >> >> > > > parameter, > > > > > > > > > > >> > >> >> > > > > as > > > > > > > > > > >> > >> >> > > > > > > > > > discussed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > above. We would need > > to > > > > > find a > > > > > > > way > > > > > > > > > to > > > > > > > > > > >> set > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > > > > direct_memory > > > > > > > > > > >> > >> >> > > > > > > > > > parameter > > > > > > > > > > >> > >> >> > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > default. We could > > start > > > > with > > > > > > 64 > > > > > > > MB > > > > > > > > > and > > > > > > > > > > >> see > > > > > > > > > > >> > >> how > > > > > > > > > > >> > >> >> it > > > > > > > > > > >> > >> >> > > goes > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > >> > >> >> > > > > > > > > practice. > > > > > > > > > > >> > >> >> > > > > > > > > > > One > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > danger I see is that > > > > setting > > > > > > > this > > > > > > > > > loo > > > > > > > > > > >> low > > > > > > > > > > >> > can > > > > > > > > > > >> > >> >> > cause a > > > > > > > > > > >> > >> >> > > > > bunch > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > additional > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GCs compared to > before > > > (we > > > > > > need > > > > > > > to > > > > > > > > > > watch > > > > > > > > > > >> > this > > > > > > > > > > >> > >> >> > > > carefully). > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 2. It is > > > > actually > > > > > > > quite > > > > > > > > > > >> simple > > > > > > > > > > >> > to > > > > > > > > > > >> > >> >> > > implement, > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > >> > >> >> > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > try > > > > > > > > > > >> > >> >> > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > segfault safe we are > > at > > > > the > > > > > > > > moment. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.2: We > > would > > > > not > > > > > > > touch > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > >> > >> >> > > > > > > > > > > > > parameter > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > at all and assume > that > > > all > > > > > the > > > > > > > > > direct > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > > allocations > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Netty do are > > infrequent > > > > > enough > > > > > > > to > > > > > > > > be > > > > > > > > > > >> > cleaned > > > > > > > > > > >> > >> up > > > > > > > > > > >> > >> >> > fast > > > > > > > > > > >> > >> >> > > > > enough > > > > > > > > > > >> > >> >> > > > > > > > > through > > > > > > > > > > >> > >> >> > > > > > > > > > > > > regular > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GC. I am not sure if > > > that > > > > > is a > > > > > > > > valid > > > > > > > > > > >> > >> assumption, > > > > > > > > > > >> > >> >> > > > though. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Best, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Stephan > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > 2:16 > > > > > > PM > > > > > > > > > > Xintong > > > > > > > > > > >> > Song > > > > > > > > > > >> > >> < > > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing > > > your > > > > > > > opinion > > > > > > > > > > Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor > of > > > > > > > alternative > > > > > > > > > 2. > > > > > > > > > > I > > > > > > > > > > >> was > > > > > > > > > > >> > >> >> > wondering > > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > > Unsafe.allocate() > > > > > for > > > > > > > > > off-heap > > > > > > > > > > >> > >> managed > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > > and > > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But > > > after > > > > > > > giving > > > > > > > > > it a > > > > > > > > > > >> > second > > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > > >> > >> >> > > > I > > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 > using > > > > direct > > > > > > > > memory > > > > > > > > > > for > > > > > > > > > > >> > >> off-heap > > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > > concern, > > > > I > > > > > > > think > > > > > > > > > what > > > > > > > > > > >> > >> proposed > > > > > > > > > > >> > >> >> in > > > > > > > > > > >> > >> >> > > this > > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > > > memory > > > > > and > > > > > > > > > network > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), > > > which > > > > > > means > > > > > > > > > they > > > > > > > > > > >> are > > > > > > > > > > >> > >> >> > practically > > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > > > > direct > > > > > > > > memory. > > > > > > > > > > The > > > > > > > > > > >> > only > > > > > > > > > > >> > >> >> parts > > > > > > > > > > >> > >> >> > of > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory > > are > > > > task > > > > > > > > > off-heap > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> and > > > > > > > > > > >> > >> >> > JVM > > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly > alternative > > 2 > > > > > > suggests > > > > > > > > to > > > > > > > > > > set > > > > > > > > > > >> the > > > > > > > > > > >> > >> JVM > > > > > > > > > > >> > >> >> max > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > 1:48 > > > > > > > PM > > > > > > > > > Till > > > > > > > > > > >> > >> Rohrmann > > > > > > > > > > >> > >> >> < > > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > > > clarification > > > > > > > > > > >> Xintong. I > > > > > > > > > > >> > >> >> > > understand > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in > > favour > > > > of > > > > > > > > option 2 > > > > > > > > > > >> > because > > > > > > > > > > >> > >> it > > > > > > > > > > >> > >> >> > makes > > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the > > > direct > > > > > > > > memory, I > > > > > > > > > > >> fear > > > > > > > > > > >> > >> that > > > > > > > > > > >> > >> >> we > > > > > > > > > > >> > >> >> > > might > > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we > > are > > > > > > > currently > > > > > > > > > in: > > > > > > > > > > >> The > > > > > > > > > > >> > >> user > > > > > > > > > > >> > >> >> > might > > > > > > > > > > >> > >> >> > > > see > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS > > and > > > > > does > > > > > > > not > > > > > > > > > know > > > > > > > > > > >> why > > > > > > > > > > >> > >> this > > > > > > > > > > >> > >> >> is > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > decrease > > > the > > > > > > > process > > > > > > > > > > memory > > > > > > > > > > >> > size > > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order > to > > > > > > > accommodate > > > > > > > > > for > > > > > > > > > > >> the > > > > > > > > > > >> > >> extra > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > decrease > > > > memory > > > > > > > > budgets > > > > > > > > > > >> which > > > > > > > > > > >> > >> are > > > > > > > > > > >> > >> >> not > > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the > overall > > > > > memory > > > > > > > > > > >> consumption. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > > 11:01 > > > > > > > > AM > > > > > > > > > > >> > Xintong > > > > > > > > > > >> > >> >> Song < > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain > > > this > > > > > > with a > > > > > > > > > > >> concrete > > > > > > > > > > >> > >> >> example > > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we > > have > > > > the > > > > > > > > > following > > > > > > > > > > >> > >> scenario. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > > > > Memory: > > > > > > 1GB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > > Memory > > > > > (Task > > > > > > > > > Off-Heap > > > > > > > > > > >> > >> Memory + > > > > > > > > > > >> > >> >> JVM > > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory > > (JVM > > > > Heap > > > > > > > > Memory, > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > Memory): > > > > 800MB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > alternative > > 2, > > > > we > > > > > > set > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > alternative > > 3, > > > > we > > > > > > set > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > > direct > > > > > > > memory > > > > > > > > > > usage > > > > > > > > > > >> of > > > > > > > > > > >> > >> Task > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed > > > 200MB, > > > > > > then > > > > > > > > > > >> > alternative 2 > > > > > > > > > > >> > >> >> and > > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > > > > Setting > > > > > > > larger > > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the > > other > > > > > > memory > > > > > > > > > pools. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > > direct > > > > > > > memory > > > > > > > > > > usage > > > > > > > > > > >> of > > > > > > > > > > >> > >> Task > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > > potentially > > > > > > > exceed > > > > > > > > > > 200MB, > > > > > > > > > > >> > then > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > > Alternative 2 > > > > > > suffers > > > > > > > > > from > > > > > > > > > > >> > >> frequent > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > >> > >> >> > > To > > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do > > is > > > to > > > > > > > modify > > > > > > > > > the > > > > > > > > > > >> > >> >> configuration > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > > Off-Heap > > > > > > Memory + > > > > > > > > JVM > > > > > > > > > > >> > >> Overhead). > > > > > > > > > > >> > >> >> > Let's > > > > > > > > > > >> > >> >> > > > say > > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct > Memory > > > to > > > > > > 250MB, > > > > > > > > > this > > > > > > > > > > >> will > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to > > 750MB, > > > > > given > > > > > > > the > > > > > > > > > > total > > > > > > > > > > >> > >> process > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > > alternative > > > > > 3, > > > > > > > > there > > > > > > > > > is > > > > > > > > > > >> no > > > > > > > > > > >> > >> >> chance of > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of > exceeding > > > the > > > > > > total > > > > > > > > > > process > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > limit, > > > > > > > > > > >> > >> >> > > > but > > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up > > all > > > > the > > > > > > > > reserved > > > > > > > > > > >> native > > > > > > > > > > >> > >> >> memory > > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > > > > Metaspace), > > > > > > > > if > > > > > > > > > > the > > > > > > > > > > >> > >> actual > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very > > close > > > to > > > > > > > 200MB, > > > > > > > > > user > > > > > > > > > > >> > >> probably > > > > > > > > > > >> > >> >> do > > > > > > > > > > >> > >> >> > > not > > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I > > think > > > > > from > > > > > > > the > > > > > > > > > > user's > > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for > alternative > > 2 > > > > may > > > > > > lead > > > > > > > > to > > > > > > > > > > >> lower > > > > > > > > > > >> > >> >> resource > > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug > 16, > > > 2019 > > > > > at > > > > > > > > 10:28 > > > > > > > > > AM > > > > > > > > > > >> Till > > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > > >> > >> >> > > < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you > > have > > > > to > > > > > > help > > > > > > > > me > > > > > > > > > > >> > >> understand > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > > > > memory > > > > > > > under > > > > > > > > > > >> > utilization > > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > Alternative > > 2: > > > > set > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. > Then > > > > there > > > > > > is > > > > > > > > the > > > > > > > > > > risk > > > > > > > > > > >> > that > > > > > > > > > > >> > >> >> this > > > > > > > > > > >> > >> >> > > size > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of > garbage > > > > > > > collection > > > > > > > > > and > > > > > > > > > > >> > >> >> potentially > > > > > > > > > > >> > >> >> > an > > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > Alternative > > 3: > > > > set > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > 2. > > > > This > > > > > > > would > > > > > > > > of > > > > > > > > > > >> course > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > > > alternative > > > > > > 2 > > > > > > > > now > > > > > > > > > > >> result > > > > > > > > > > >> > >> in an > > > > > > > > > > >> > >> >> > > under > > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > > > > alternative > > > > > > > 3? > > > > > > > > > If > > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > memory > > > size > > > > > and > > > > > > > we > > > > > > > > > use > > > > > > > > > > >> only > > > > > > > > > > >> > >> >> little, > > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > 3 > > > > > results > > > > > > in > > > > > > > > > > memory > > > > > > > > > > >> > under > > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug > > 13, > > > > 2019 > > > > > > at > > > > > > > > 4:19 > > > > > > > > > > PM > > > > > > > > > > >> > Yang > > > > > > > > > > >> > >> >> Wang < > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > > xintong,till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native > and > > > > > Direct > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point > is > > > > > setting > > > > > > a > > > > > > > > very > > > > > > > > > > >> large > > > > > > > > > > >> > >> max > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > differentiate > > > > > direct > > > > > > > and > > > > > > > > > > >> native > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > >> > >> >> > If > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > > memory > > > > and > > > > > > > > > framework > > > > > > > > > > >> > direct > > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in > > favor > > > of > > > > > > > setting > > > > > > > > > > >> direct > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > with > > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > > > Calculation > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree > with > > > > > > xintong. > > > > > > > > For > > > > > > > > > > Yarn > > > > > > > > > > >> > and > > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > > >> > >> >> > > > need > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configurations > > > > in > > > > > > > client > > > > > > > > > to > > > > > > > > > > >> avoid > > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > > > > master. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong > > Song < > > > > > > > > > > >> > >> [hidden email] > > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks > for > > > > > > replying, > > > > > > > > > Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > > > MemorySegment, I > > > > > > > > > > think > > > > > > > > > > >> > you > > > > > > > > > > >> > >> are > > > > > > > > > > >> > >> >> > > right > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in > > the > > > > > scope > > > > > > > of > > > > > > > > > this > > > > > > > > > > >> > FLIP. > > > > > > > > > > >> > >> >> This > > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configure > > > > memory > > > > > > > pools > > > > > > > > > for > > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > > > consumers > > > > > > use > > > > > > > > it. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > direct > > > > > > > memory, I > > > > > > > > > > think > > > > > > > > > > >> > >> >> > alternative > > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > reservation > > > > > issue > > > > > > > that > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > >> > >> >> > > does, > > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using > > memory > > > > at > > > > > > the > > > > > > > > > > >> container > > > > > > > > > > >> > >> level, > > > > > > > > > > >> > >> >> > > which > > > > > > > > > > >> > >> >> > > > is > > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both > "Task > > > > > > Off-Heap > > > > > > > > > > Memory" > > > > > > > > > > >> and > > > > > > > > > > >> > >> "JVM > > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > alternative > > > 2, > > > > > > users > > > > > > > > > might > > > > > > > > > > >> > >> configure > > > > > > > > > > >> > >> >> > them > > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to > > > avoid > > > > > > > getting > > > > > > > > a > > > > > > > > > > >> direct > > > > > > > > > > >> > >> OOM. > > > > > > > > > > >> > >> >> For > > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > OOM, > > > so > > > > > > they > > > > > > > > may > > > > > > > > > > not > > > > > > > > > > >> > >> config > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > two > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > > consequences > > > > > > are > > > > > > > > > risks > > > > > > > > > > >> of > > > > > > > > > > >> > >> >> overall > > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank > you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong > > Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, > > Aug > > > > 13, > > > > > > 2019 > > > > > > > > at > > > > > > > > > > >> 9:39 AM > > > > > > > > > > >> > >> Till > > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > > > > proposing > > > > > > > > > > this > > > > > > > > > > >> > FLIP > > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in > > > all I > > > > > > think > > > > > > > > it > > > > > > > > > > >> already > > > > > > > > > > >> > >> >> looks > > > > > > > > > > >> > >> >> > > quite > > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > question > > > > about > > > > > > > > > > allocating > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > strictly > > > > > > necessary > > > > > > > > to > > > > > > > > > do > > > > > > > > > > >> in > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > context > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > done > > > as a > > > > > > > follow > > > > > > > > > up? > > > > > > > > > > >> > Without > > > > > > > > > > >> > >> >> > knowing > > > > > > > > > > >> > >> >> > > > all > > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that > we > > > > would > > > > > > > widen > > > > > > > > > the > > > > > > > > > > >> scope > > > > > > > > > > >> > >> of > > > > > > > > > > >> > >> >> this > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to > touch > > > all > > > > > the > > > > > > > > > > existing > > > > > > > > > > >> > call > > > > > > > > > > >> > >> >> sites > > > > > > > > > > >> > >> >> > of > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > > > segments > > > > > > > > (this > > > > > > > > > > >> should > > > > > > > > > > >> > >> >> mainly > > > > > > > > > > >> > >> >> > be > > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > memory > > > > > > > > reservation > > > > > > > > > > >> call > > > > > > > > > > >> > to > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this > > and I > > > > > would > > > > > > > > hope > > > > > > > > > > that > > > > > > > > > > >> > >> this is > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > streaming > > > > job > > > > > > > would > > > > > > > > > have > > > > > > > > > > >> with > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Concerning > > > > the > > > > > > > > second > > > > > > > > > > open > > > > > > > > > > >> > >> >> question > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > > memory > > > > > > > > limit, I > > > > > > > > > > >> would > > > > > > > > > > >> > >> also > > > > > > > > > > >> > >> >> be > > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > leaving > > it > > > > > open > > > > > > > > would > > > > > > > > > be > > > > > > > > > > >> > best. > > > > > > > > > > >> > >> My > > > > > > > > > > >> > >> >> > > concern > > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in > a > > > > > similar > > > > > > > > > > situation > > > > > > > > > > >> as > > > > > > > > > > >> > we > > > > > > > > > > >> > >> >> are > > > > > > > > > > >> > >> >> > now > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > > different > > > > > > > memory > > > > > > > > > > pools > > > > > > > > > > >> > are > > > > > > > > > > >> > >> not > > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > > different > > > > > > pool, > > > > > > > > then > > > > > > > > > > it > > > > > > > > > > >> is > > > > > > > > > > >> > >> quite > > > > > > > > > > >> > >> >> > hard > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > process > > to > > > > get > > > > > > > > killed > > > > > > > > > > for > > > > > > > > > > >> > using > > > > > > > > > > >> > >> >> too > > > > > > > > > > >> > >> >> > > much > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead > to > > a > > > > > > similar > > > > > > > > > > >> situation > > > > > > > > > > >> > >> what > > > > > > > > > > >> > >> >> we > > > > > > > > > > >> > >> >> > > have > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > setting > > a > > > > sane > > > > > > > > default > > > > > > > > > > >> value > > > > > > > > > > >> > >> for > > > > > > > > > > >> > >> >> max > > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option > > to > > > > > > increase > > > > > > > > it > > > > > > > > > if > > > > > > > > > > >> he > > > > > > > > > > >> > >> runs > > > > > > > > > > >> > >> >> into > > > > > > > > > > >> > >> >> > > an > > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > > how > > > > > > > would > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > >> > >> >> > lead > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > 3 > > > > > > > where > > > > > > > > we > > > > > > > > > > set > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> direct > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On > Fri, > > > Aug > > > > 9, > > > > > > > 2019 > > > > > > > > at > > > > > > > > > > >> 9:12 > > > > > > > > > > >> > AM > > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > the > > > > > > > > > > feedback, > > > > > > > > > > >> > >> Yang. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > > your > > > > > > > > > > comments: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > *Native > > > > and > > > > > > > Direct > > > > > > > > > > >> Memory* > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > think > > > > > > setting > > > > > > > a > > > > > > > > > very > > > > > > > > > > >> > large > > > > > > > > > > >> > >> max > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > > > > sides. > > > > > > > E.g., > > > > > > > > we > > > > > > > > > > do > > > > > > > > > > >> not > > > > > > > > > > >> > >> >> worry > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > > > allocate > > > > > > > > managed > > > > > > > > > / > > > > > > > > > > >> > network > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > However, > > > > > there > > > > > > > are > > > > > > > > > > also > > > > > > > > > > >> > some > > > > > > > > > > >> > >> >> down > > > > > > > > > > >> > >> >> > > sides > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > One > > > > > > thing I > > > > > > > > can > > > > > > > > > > >> think > > > > > > > > > > >> > >> of is > > > > > > > > > > >> > >> >> > that > > > > > > > > > > >> > >> >> > > > if > > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > killed > > > > > due > > > > > > to > > > > > > > > > > >> overusing > > > > > > > > > > >> > >> >> memory, > > > > > > > > > > >> > >> >> > it > > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > of > > > the > > > > > > memory > > > > > > > > is > > > > > > > > > > >> > overused. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > > > Another > > > > > > > down > > > > > > > > > side > > > > > > > > > > >> is > > > > > > > > > > >> > >> that > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > memory > > > > > > > > > > limit, > > > > > > > > > > >> > >> because > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > means > > > > we > > > > > > kind > > > > > > > > of > > > > > > > > > > >> relay > > > > > > > > > > >> > on > > > > > > > > > > >> > >> >> heap > > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > > That > > > > > > > > > could > > > > > > > > > > >> be a > > > > > > > > > > >> > >> >> problem > > > > > > > > > > >> > >> >> > in > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > usage > > > > but > > > > > > not > > > > > > > > > > enough > > > > > > > > > > >> > heap > > > > > > > > > > >> > >> >> > activity > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Maybe > > > you > > > > > can > > > > > > > > share > > > > > > > > > > your > > > > > > > > > > >> > >> reasons > > > > > > > > > > >> > >> >> > for > > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if > > there > > > > are > > > > > > > > > anything > > > > > > > > > > >> else > > > > > > > > > > >> > I > > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > > Calculation* > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If > > there > > > > is > > > > > > any > > > > > > > > > > conflict > > > > > > > > > > >> > >> between > > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > > specified, > > > > > > > > > > I > > > > > > > > > > >> > >> think we > > > > > > > > > > >> > >> >> > > should > > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > think > > > > > doing > > > > > > > > > checking > > > > > > > > > > >> on > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > client > > > > > > > > > > >> > >> >> > > > side > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s > we > > > can > > > > > > > > discover > > > > > > > > > > the > > > > > > > > > > >> > >> problem > > > > > > > > > > >> > >> >> > > before > > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > > > always > > > > a > > > > > > good > > > > > > > > > > thing. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But > we > > > can > > > > > not > > > > > > > > only > > > > > > > > > > >> rely on > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > client > > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > > cluster > > > > > > > > > > >> > >> TaskManagers > > > > > > > > > > >> > >> >> on > > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > > and > > > > > > > > > the > > > > > > > > > > >> > client > > > > > > > > > > >> > >> >> does > > > > > > > > > > >> > >> >> > > see > > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What > > do > > > > you > > > > > > > think? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thank > > > you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On > > Thu, > > > > Aug > > > > > 8, > > > > > > > > 2019 > > > > > > > > > at > > > > > > > > > > >> 5:09 > > > > > > > > > > >> > >> PM > > > > > > > > > > >> > >> >> Yang > > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > xintong, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > for > > > > > > > your > > > > > > > > > > >> detailed > > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > > it > > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > >> > more > > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > just > > > > have > > > > > > few > > > > > > > > > > >> questions > > > > > > > > > > >> > >> about > > > > > > > > > > >> > >> >> it. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Native > > > > > > > and > > > > > > > > > > Direct > > > > > > > > > > >> > >> Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We > > do > > > > not > > > > > > > > > > >> differentiate > > > > > > > > > > >> > >> user > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > included > > > > > in > > > > > > > task > > > > > > > > > > >> off-heap > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Memory > > > > > > > > > > >> Calculation > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If > > the > > > > sum > > > > > > of > > > > > > > > and > > > > > > > > > > >> > >> fine-grained > > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > > > larger > > > > > > than > > > > > > > > > total > > > > > > > > > > >> > >> process > > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we > > > need > > > > to > > > > > > > check > > > > > > > > > the > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > Song < > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > everyone, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > We > > > > would > > > > > > > like > > > > > > > > to > > > > > > > > > > >> start > > > > > > > > > > >> > a > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > > design > > > > > > > > > > "Memory > > > > > > > > > > >> > >> >> Management > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > updates > > > > > > > > > from > > > > > > > > > > >> > >> follow-up > > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > This > > > > > FLIP > > > > > > > > > > addresses > > > > > > > > > > >> > >> several > > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Different > > > > > > > > > > >> > >> configuration > > > > > > > > > > >> > >> >> > for > > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Complex > > > > > > > > and > > > > > > > > > > >> > >> difficult > > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > Complicated, > > > > > > > > > > >> > >> uncertain > > > > > > > > > > >> > >> >> and > > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > > changes > > > > > > > to > > > > > > > > > > solve > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > problems > > > > > > > > > > >> > >> >> > > > can > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Extend > > > > > > > > > memory > > > > > > > > > > >> > >> manager > > > > > > > > > > >> > >> >> to > > > > > > > > > > >> > >> >> > > also > > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Modify > > > > > > > > how > > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > >> reservations > > > > > > > > > > >> > >> and > > > > > > > > > > >> > >> >> > pools. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Simplify > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > > find > > > > > > > > more > > > > > > > > > > >> > details > > > > > > > > > > >> > >> in > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > > note > > > > > > > > > that > > > > > > > > > > >> the > > > > > > > > > > >> > >> early > > > > > > > > > > >> > >> >> > > design > > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > > to > > > > > > > > > > have > > > > > > > > > > >> the > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > > forward > > > > > > > > > to > > > > > > > > > > >> your > > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, 2019 > > at > > > > 2:16 > > > > > > PM > > > > > > > > > > Xintong > > > > > > > > > > >> > Song > > > > > > > > > > >> > >> < > > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for sharing > > > your > > > > > > > opinion > > > > > > > > > > Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in favor > of > > > > > > > alternative > > > > > > > > > 2. > > > > > > > > > > I > > > > > > > > > > >> was > > > > > > > > > > >> > >> >> > wondering > > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > > Unsafe.allocate() > > > > > for > > > > > > > > > off-heap > > > > > > > > > > >> > >> managed > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > > and > > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. But > > > after > > > > > > > giving > > > > > > > > > it a > > > > > > > > > > >> > second > > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > > >> > >> >> > > > I > > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 > using > > > > direct > > > > > > > > memory > > > > > > > > > > for > > > > > > > > > > >> > >> off-heap > > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > > concern, > > > > I > > > > > > > think > > > > > > > > > what > > > > > > > > > > >> > >> proposed > > > > > > > > > > >> > >> >> in > > > > > > > > > > >> > >> >> > > this > > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > > > memory > > > > > and > > > > > > > > > network > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate(), > > > which > > > > > > means > > > > > > > > > they > > > > > > > > > > >> are > > > > > > > > > > >> > >> >> > practically > > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM max > > > > direct > > > > > > > > memory. > > > > > > > > > > The > > > > > > > > > > >> > only > > > > > > > > > > >> > >> >> parts > > > > > > > > > > >> > >> >> > of > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct memory > > are > > > > task > > > > > > > > > off-heap > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> and > > > > > > > > > > >> > >> >> > JVM > > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly > alternative > > 2 > > > > > > suggests > > > > > > > > to > > > > > > > > > > set > > > > > > > > > > >> the > > > > > > > > > > >> > >> JVM > > > > > > > > > > >> > >> >> max > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > 1:48 > > > > > > > PM > > > > > > > > > Till > > > > > > > > > > >> > >> Rohrmann > > > > > > > > > > >> > >> >> < > > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > > > clarification > > > > > > > > > > >> Xintong. I > > > > > > > > > > >> > >> >> > > understand > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in > > favour > > > > of > > > > > > > > option 2 > > > > > > > > > > >> > because > > > > > > > > > > >> > >> it > > > > > > > > > > >> > >> >> > makes > > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit the > > > direct > > > > > > > > memory, I > > > > > > > > > > >> fear > > > > > > > > > > >> > >> that > > > > > > > > > > >> > >> >> we > > > > > > > > > > >> > >> >> > > might > > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as we > > are > > > > > > > currently > > > > > > > > > in: > > > > > > > > > > >> The > > > > > > > > > > >> > >> user > > > > > > > > > > >> > >> >> > might > > > > > > > > > > >> > >> >> > > > see > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the OS > > and > > > > > does > > > > > > > not > > > > > > > > > know > > > > > > > > > > >> why > > > > > > > > > > >> > >> this > > > > > > > > > > >> > >> >> is > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > decrease > > > the > > > > > > > process > > > > > > > > > > memory > > > > > > > > > > >> > size > > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in order > to > > > > > > > accommodate > > > > > > > > > for > > > > > > > > > > >> the > > > > > > > > > > >> > >> extra > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > decrease > > > > memory > > > > > > > > budgets > > > > > > > > > > >> which > > > > > > > > > > >> > >> are > > > > > > > > > > >> > >> >> not > > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the > overall > > > > > memory > > > > > > > > > > >> consumption. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > > 11:01 > > > > > > > > AM > > > > > > > > > > >> > Xintong > > > > > > > > > > >> > >> >> Song < > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me explain > > > this > > > > > > with a > > > > > > > > > > >> concrete > > > > > > > > > > >> > >> >> example > > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we > > have > > > > the > > > > > > > > > following > > > > > > > > > > >> > >> scenario. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total Process > > > > Memory: > > > > > > 1GB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > > Memory > > > > > (Task > > > > > > > > > Off-Heap > > > > > > > > > > >> > >> Memory + > > > > > > > > > > >> > >> >> JVM > > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory > > (JVM > > > > Heap > > > > > > > > Memory, > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > Memory): > > > > 800MB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > alternative > > 2, > > > > we > > > > > > set > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > alternative > > 3, > > > > we > > > > > > set > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say 1TB. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > > direct > > > > > > > memory > > > > > > > > > > usage > > > > > > > > > > >> of > > > > > > > > > > >> > >> Task > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not exceed > > > 200MB, > > > > > > then > > > > > > > > > > >> > alternative 2 > > > > > > > > > > >> > >> >> and > > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same utility. > > > > Setting > > > > > > > larger > > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the > > other > > > > > > memory > > > > > > > > > pools. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the actual > > > direct > > > > > > > memory > > > > > > > > > > usage > > > > > > > > > > >> of > > > > > > > > > > >> > >> Task > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > > potentially > > > > > > > exceed > > > > > > > > > > 200MB, > > > > > > > > > > >> > then > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > > Alternative 2 > > > > > > suffers > > > > > > > > > from > > > > > > > > > > >> > >> frequent > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > >> > >> >> > > To > > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can do > > is > > > to > > > > > > > modify > > > > > > > > > the > > > > > > > > > > >> > >> >> configuration > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > > Off-Heap > > > > > > Memory + > > > > > > > > JVM > > > > > > > > > > >> > >> Overhead). > > > > > > > > > > >> > >> >> > Let's > > > > > > > > > > >> > >> >> > > > say > > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct > Memory > > > to > > > > > > 250MB, > > > > > > > > > this > > > > > > > > > > >> will > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to > > 750MB, > > > > > given > > > > > > > the > > > > > > > > > > total > > > > > > > > > > >> > >> process > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > > alternative > > > > > 3, > > > > > > > > there > > > > > > > > > is > > > > > > > > > > >> no > > > > > > > > > > >> > >> >> chance of > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of > exceeding > > > the > > > > > > total > > > > > > > > > > process > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > limit, > > > > > > > > > > >> > >> >> > > > but > > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use up > > all > > > > the > > > > > > > > reserved > > > > > > > > > > >> native > > > > > > > > > > >> > >> >> memory > > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, JVM > > > > > > Metaspace), > > > > > > > > if > > > > > > > > > > the > > > > > > > > > > >> > >> actual > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very > > close > > > to > > > > > > > 200MB, > > > > > > > > > user > > > > > > > > > > >> > >> probably > > > > > > > > > > >> > >> >> do > > > > > > > > > > >> > >> >> > > not > > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I > > think > > > > > from > > > > > > > the > > > > > > > > > > user's > > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for > alternative > > 2 > > > > may > > > > > > lead > > > > > > > > to > > > > > > > > > > >> lower > > > > > > > > > > >> > >> >> resource > > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 3. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug > 16, > > > 2019 > > > > > at > > > > > > > > 10:28 > > > > > > > > > AM > > > > > > > > > > >> Till > > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > > >> > >> >> > > < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess you > > have > > > > to > > > > > > help > > > > > > > > me > > > > > > > > > > >> > >> understand > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative 2 > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt to > > > > memory > > > > > > > under > > > > > > > > > > >> > utilization > > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > Alternative > > 2: > > > > set > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. > Then > > > > there > > > > > > is > > > > > > > > the > > > > > > > > > > risk > > > > > > > > > > >> > that > > > > > > > > > > >> > >> >> this > > > > > > > > > > >> > >> >> > > size > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of > garbage > > > > > > > collection > > > > > > > > > and > > > > > > > > > > >> > >> >> potentially > > > > > > > > > > >> > >> >> > an > > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > Alternative > > 3: > > > > set > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > 2. > > > > This > > > > > > > would > > > > > > > > of > > > > > > > > > > >> course > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > > > alternative > > > > > > 2 > > > > > > > > now > > > > > > > > > > >> result > > > > > > > > > > >> > >> in an > > > > > > > > > > >> > >> >> > > under > > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared to > > > > > > alternative > > > > > > > 3? > > > > > > > > > If > > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > memory > > > size > > > > > and > > > > > > > we > > > > > > > > > use > > > > > > > > > > >> only > > > > > > > > > > >> > >> >> little, > > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > 3 > > > > > results > > > > > > in > > > > > > > > > > memory > > > > > > > > > > >> > under > > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, Aug > > 13, > > > > 2019 > > > > > > at > > > > > > > > 4:19 > > > > > > > > > > PM > > > > > > > > > > >> > Yang > > > > > > > > > > >> > >> >> Wang < > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > > xintong,till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native > and > > > > > Direct > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point > is > > > > > setting > > > > > > a > > > > > > > > very > > > > > > > > > > >> large > > > > > > > > > > >> > >> max > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > differentiate > > > > > direct > > > > > > > and > > > > > > > > > > >> native > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > >> > >> >> > If > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory,including > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > > memory > > > > and > > > > > > > > > framework > > > > > > > > > > >> > direct > > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in > > favor > > > of > > > > > > > setting > > > > > > > > > > >> direct > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > with > > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > > > Calculation > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree > with > > > > > > xintong. > > > > > > > > For > > > > > > > > > > Yarn > > > > > > > > > > >> > and > > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > > >> > >> >> > > > need > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configurations > > > > in > > > > > > > client > > > > > > > > > to > > > > > > > > > > >> avoid > > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the flink > > > > master. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong > > Song < > > > > > > > > > > >> > >> [hidden email] > > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks > for > > > > > > replying, > > > > > > > > > Till. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > > > MemorySegment, I > > > > > > > > > > think > > > > > > > > > > >> > you > > > > > > > > > > >> > >> are > > > > > > > > > > >> > >> >> > > right > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue in > > the > > > > > scope > > > > > > > of > > > > > > > > > this > > > > > > > > > > >> > FLIP. > > > > > > > > > > >> > >> >> This > > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configure > > > > memory > > > > > > > pools > > > > > > > > > for > > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > > > consumers > > > > > > use > > > > > > > > it. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > direct > > > > > > > memory, I > > > > > > > > > > think > > > > > > > > > > >> > >> >> > alternative > > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > reservation > > > > > issue > > > > > > > that > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > >> > >> >> > > does, > > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using > > memory > > > > at > > > > > > the > > > > > > > > > > >> container > > > > > > > > > > >> > >> level, > > > > > > > > > > >> > >> >> > > which > > > > > > > > > > >> > >> >> > > > is > > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both > "Task > > > > > > Off-Heap > > > > > > > > > > Memory" > > > > > > > > > > >> and > > > > > > > > > > >> > >> "JVM > > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > alternative > > > 2, > > > > > > users > > > > > > > > > might > > > > > > > > > > >> > >> configure > > > > > > > > > > >> > >> >> > them > > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just to > > > avoid > > > > > > > getting > > > > > > > > a > > > > > > > > > > >> direct > > > > > > > > > > >> > >> OOM. > > > > > > > > > > >> > >> >> For > > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > OOM, > > > so > > > > > > they > > > > > > > > may > > > > > > > > > > not > > > > > > > > > > >> > >> config > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > two > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > > consequences > > > > > > are > > > > > > > > > risks > > > > > > > > > > >> of > > > > > > > > > > >> > >> >> overall > > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank > you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong > > Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On Tue, > > Aug > > > > 13, > > > > > > 2019 > > > > > > > > at > > > > > > > > > > >> 9:39 AM > > > > > > > > > > >> > >> Till > > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > > > > proposing > > > > > > > > > > this > > > > > > > > > > >> > FLIP > > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All in > > > all I > > > > > > think > > > > > > > > it > > > > > > > > > > >> already > > > > > > > > > > >> > >> >> looks > > > > > > > > > > >> > >> >> > > quite > > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > question > > > > about > > > > > > > > > > allocating > > > > > > > > > > >> > >> memory > > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > strictly > > > > > > necessary > > > > > > > > to > > > > > > > > > do > > > > > > > > > > >> in > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > context > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > done > > > as a > > > > > > > follow > > > > > > > > > up? > > > > > > > > > > >> > Without > > > > > > > > > > >> > >> >> > knowing > > > > > > > > > > >> > >> >> > > > all > > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that > we > > > > would > > > > > > > widen > > > > > > > > > the > > > > > > > > > > >> scope > > > > > > > > > > >> > >> of > > > > > > > > > > >> > >> >> this > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to > touch > > > all > > > > > the > > > > > > > > > > existing > > > > > > > > > > >> > call > > > > > > > > > > >> > >> >> sites > > > > > > > > > > >> > >> >> > of > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > > > segments > > > > > > > > (this > > > > > > > > > > >> should > > > > > > > > > > >> > >> >> mainly > > > > > > > > > > >> > >> >> > be > > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > memory > > > > > > > > reservation > > > > > > > > > > >> call > > > > > > > > > > >> > to > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this > > and I > > > > > would > > > > > > > > hope > > > > > > > > > > that > > > > > > > > > > >> > >> this is > > > > > > > > > > >> > >> >> > the > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > streaming > > > > job > > > > > > > would > > > > > > > > > have > > > > > > > > > > >> with > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Concerning > > > > the > > > > > > > > second > > > > > > > > > > open > > > > > > > > > > >> > >> >> question > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > > memory > > > > > > > > limit, I > > > > > > > > > > >> would > > > > > > > > > > >> > >> also > > > > > > > > > > >> > >> >> be > > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > leaving > > it > > > > > open > > > > > > > > would > > > > > > > > > be > > > > > > > > > > >> > best. > > > > > > > > > > >> > >> My > > > > > > > > > > >> > >> >> > > concern > > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be in > a > > > > > similar > > > > > > > > > > situation > > > > > > > > > > >> as > > > > > > > > > > >> > we > > > > > > > > > > >> > >> >> are > > > > > > > > > > >> > >> >> > now > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > > different > > > > > > > memory > > > > > > > > > > pools > > > > > > > > > > >> > are > > > > > > > > > > >> > >> not > > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > > different > > > > > > pool, > > > > > > > > then > > > > > > > > > > it > > > > > > > > > > >> is > > > > > > > > > > >> > >> quite > > > > > > > > > > >> > >> >> > hard > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > process > > to > > > > get > > > > > > > > killed > > > > > > > > > > for > > > > > > > > > > >> > using > > > > > > > > > > >> > >> >> too > > > > > > > > > > >> > >> >> > > much > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead > to > > a > > > > > > similar > > > > > > > > > > >> situation > > > > > > > > > > >> > >> what > > > > > > > > > > >> > >> >> we > > > > > > > > > > >> > >> >> > > have > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > setting > > a > > > > sane > > > > > > > > default > > > > > > > > > > >> value > > > > > > > > > > >> > >> for > > > > > > > > > > >> > >> >> max > > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > option > > to > > > > > > increase > > > > > > > > it > > > > > > > > > if > > > > > > > > > > >> he > > > > > > > > > > >> > >> runs > > > > > > > > > > >> > >> >> into > > > > > > > > > > >> > >> >> > > an > > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > > how > > > > > > > would > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > >> > >> >> > lead > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > 3 > > > > > > > where > > > > > > > > we > > > > > > > > > > set > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> direct > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On > Fri, > > > Aug > > > > 9, > > > > > > > 2019 > > > > > > > > at > > > > > > > > > > >> 9:12 > > > > > > > > > > >> > AM > > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > the > > > > > > > > > > feedback, > > > > > > > > > > >> > >> Yang. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > > your > > > > > > > > > > comments: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > *Native > > > > and > > > > > > > Direct > > > > > > > > > > >> Memory* > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > think > > > > > > setting > > > > > > > a > > > > > > > > > very > > > > > > > > > > >> > large > > > > > > > > > > >> > >> max > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > good > > > > sides. > > > > > > > E.g., > > > > > > > > we > > > > > > > > > > do > > > > > > > > > > >> not > > > > > > > > > > >> > >> >> worry > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > > > allocate > > > > > > > > managed > > > > > > > > > / > > > > > > > > > > >> > network > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > >> > >> >> > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() . > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > However, > > > > > there > > > > > > > are > > > > > > > > > > also > > > > > > > > > > >> > some > > > > > > > > > > >> > >> >> down > > > > > > > > > > >> > >> >> > > sides > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > One > > > > > > thing I > > > > > > > > can > > > > > > > > > > >> think > > > > > > > > > > >> > >> of is > > > > > > > > > > >> > >> >> > that > > > > > > > > > > >> > >> >> > > > if > > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > killed > > > > > due > > > > > > to > > > > > > > > > > >> overusing > > > > > > > > > > >> > >> >> memory, > > > > > > > > > > >> > >> >> > it > > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > of > > > the > > > > > > memory > > > > > > > > is > > > > > > > > > > >> > overused. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > - > > > > Another > > > > > > > down > > > > > > > > > side > > > > > > > > > > >> is > > > > > > > > > > >> > >> that > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > memory > > > > > > > > > > limit, > > > > > > > > > > >> > >> because > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > means > > > > we > > > > > > kind > > > > > > > > of > > > > > > > > > > >> relay > > > > > > > > > > >> > on > > > > > > > > > > >> > >> >> heap > > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > > That > > > > > > > > > could > > > > > > > > > > >> be a > > > > > > > > > > >> > >> >> problem > > > > > > > > > > >> > >> >> > in > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > usage > > > > but > > > > > > not > > > > > > > > > > enough > > > > > > > > > > >> > heap > > > > > > > > > > >> > >> >> > activity > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Maybe > > > you > > > > > can > > > > > > > > share > > > > > > > > > > your > > > > > > > > > > >> > >> reasons > > > > > > > > > > >> > >> >> > for > > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if > > there > > > > are > > > > > > > > > anything > > > > > > > > > > >> else > > > > > > > > > > >> > I > > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > > Calculation* > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If > > there > > > > is > > > > > > any > > > > > > > > > > conflict > > > > > > > > > > >> > >> between > > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > > specified, > > > > > > > > > > I > > > > > > > > > > >> > >> think we > > > > > > > > > > >> > >> >> > > should > > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > think > > > > > doing > > > > > > > > > checking > > > > > > > > > > >> on > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > client > > > > > > > > > > >> > >> >> > > > side > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > K8s > we > > > can > > > > > > > > discover > > > > > > > > > > the > > > > > > > > > > >> > >> problem > > > > > > > > > > >> > >> >> > > before > > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > > > always > > > > a > > > > > > good > > > > > > > > > > thing. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > But > we > > > can > > > > > not > > > > > > > > only > > > > > > > > > > >> rely on > > > > > > > > > > >> > >> the > > > > > > > > > > >> > >> >> > > client > > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > > cluster > > > > > > > > > > >> > >> TaskManagers > > > > > > > > > > >> > >> >> on > > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > > and > > > > > > > > > the > > > > > > > > > > >> > client > > > > > > > > > > >> > >> >> does > > > > > > > > > > >> > >> >> > > see > > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > What > > do > > > > you > > > > > > > think? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thank > > > you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On > > Thu, > > > > Aug > > > > > 8, > > > > > > > > 2019 > > > > > > > > > at > > > > > > > > > > >> 5:09 > > > > > > > > > > >> > >> PM > > > > > > > > > > >> > >> >> Yang > > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > xintong, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > for > > > > > > > your > > > > > > > > > > >> detailed > > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > > it > > > > > > > > > will > > > > > > > > > > be > > > > > > > > > > >> > more > > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > just > > > > have > > > > > > few > > > > > > > > > > >> questions > > > > > > > > > > >> > >> about > > > > > > > > > > >> > >> >> it. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Native > > > > > > > and > > > > > > > > > > Direct > > > > > > > > > > >> > >> Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > We > > do > > > > not > > > > > > > > > > >> differentiate > > > > > > > > > > >> > >> user > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > included > > > > > in > > > > > > > task > > > > > > > > > > >> off-heap > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > Memory > > > > > > > > > > >> Calculation > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If > > the > > > > sum > > > > > > of > > > > > > > > and > > > > > > > > > > >> > >> fine-grained > > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > > > larger > > > > > > than > > > > > > > > > total > > > > > > > > > > >> > >> process > > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > we > > > need > > > > to > > > > > > > check > > > > > > > > > the > > > > > > > > > > >> > memory > > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > Song < > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > everyone, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > We > > > > would > > > > > > > like > > > > > > > > to > > > > > > > > > > >> start > > > > > > > > > > >> > a > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > for > > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > > design > > > > > > > > > > "Memory > > > > > > > > > > >> > >> >> Management > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > updates > > > > > > > > > from > > > > > > > > > > >> > >> follow-up > > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > This > > > > > FLIP > > > > > > > > > > addresses > > > > > > > > > > >> > >> several > > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Different > > > > > > > > > > >> > >> configuration > > > > > > > > > > >> > >> >> > for > > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Complex > > > > > > > > and > > > > > > > > > > >> > >> difficult > > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > Complicated, > > > > > > > > > > >> > >> uncertain > > > > > > > > > > >> > >> >> and > > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > > changes > > > > > > > to > > > > > > > > > > solve > > > > > > > > > > >> > the > > > > > > > > > > >> > >> >> > problems > > > > > > > > > > >> > >> >> > > > can > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Extend > > > > > > > > > memory > > > > > > > > > > >> > >> manager > > > > > > > > > > >> > >> >> to > > > > > > > > > > >> > >> >> > > also > > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Modify > > > > > > > > how > > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > >> reservations > > > > > > > > > > >> > >> and > > > > > > > > > > >> > >> >> > pools. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Simplify > > > > > > > > > > memory > > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > > find > > > > > > > > more > > > > > > > > > > >> > details > > > > > > > > > > >> > >> in > > > > > > > > > > >> > >> >> the > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > > note > > > > > > > > > that > > > > > > > > > > >> the > > > > > > > > > > >> > >> early > > > > > > > > > > >> > >> >> > > design > > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > > to > > > > > > > > > > have > > > > > > > > > > >> the > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > > forward > > > > > > > > > to > > > > > > > > > > >> your > > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > you~ > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > Song > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
|
|
Fair enough.
Thank you~ Xintong Song On Mon, Sep 16, 2019 at 7:28 PM Till Rohrmann <[hidden email]> wrote: > Yes this could be the case. But from the point of the slot, it doesn't > really matter to it because it is abstracted away and can be added at a > later time when really needed. > > Cheers, > Till > > On Mon, Sep 16, 2019 at 11:42 AM Xintong Song <[hidden email]> > wrote: > > > @Till > > > > Just trying to understand. Are you suggesting that we have both a task > > executor level memory manager, and multiple slot level memory managers > with > > in the task executor level one? (Or what ever the components are called) > > > > If that is what you mean, then it makes a lot sense to me. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Sep 16, 2019 at 5:30 PM Till Rohrmann <[hidden email]> > > wrote: > > > > > I think conceptually it makes sense to have a MemoryManager per slot. > As > > > Andrey said, it should not add any noticeable overhead. Moreover, I > think > > > this model does not contradict any of the mentioned future > improvements. > > > For example, a per slot MemoryManager could still be implemented such > > that > > > the TaskExecutor's MemoryManager keeps a reference to it and can > request > > to > > > give up/assign more memory segments in case of revocable memory. The > > > benefit I see is a better abstraction and hence a more targeted > component > > > which will be easier to maintain and evolve. > > > > > > Cheers, > > > Till > > > > > > On Mon, Sep 16, 2019 at 11:23 AM Xintong Song <[hidden email]> > > > wrote: > > > > > > > I would say both ways (one unified memory manager per task executor, > or > > > one > > > > isolated memory manager per slot) should work well at the moment. > I've > > > not > > > > yet think this through very detailedly, but just from what I can > think > > of > > > > at the moment, I'm in favor of having one memory manager per task > > > executor, > > > > for the following reasons: > > > > > > > > - We need the memory manager to understand quota of each operator > > anyway > > > > (with the provided fractions), for limiting the memory usage of > > > > un-revocable operators. It shouldn't be much more complicated for the > > > > memory manager to isolate memory across slots. Actually I'm wondering > > > > wether it requires more efforts at all compare to having one memory > > > manager > > > > per slot, where we also need to handle the creation and destroying of > > > > memory managers. > > > > > > > > - It is true that we have no plan for segments caching at the moment. > > > But I > > > > would say it is a good opportunity for future improvement. E.g., we > may > > > > have some "lazy de-allocation" strategy that only de-allocate > segments > > if > > > > the remaining available memory quota cannot satisfy new coming > > > reservation > > > > requests. I understand that we should not complicate the current > design > > > for > > > > any "potential future features". My concern is that we make efforts > > > > changing from one memory manager per task executor to one per slot, > > with > > > > the benefit (easier to isolate memory across slots) that is uncertain > > to > > > > me, and later discover that we may want to change it back for further > > > > optimizations. > > > > > > > > Anyway, I do not have strong objection against having one memory > > manager > > > > per slot. Just raising my concerns. > > > > > > > > Thank you~ > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 4:16 PM Andrey Zagrebin < > [hidden email]> > > > > wrote: > > > > > > > > > @Xintong > > > > > > > > > > Just to understand, what kind of overhead do you mean by creating > the > > > > > memory manager for every dynamic slot? > > > > > We agreed not to preallocate segments anymore and there was no plan > > to > > > do > > > > > any segment caching at the moment, AFAIK. > > > > > This means the segments are just garbage collected upon release and > > not > > > > > reused. > > > > > If we decide to go with caching or revoking later and share pools > > among > > > > > slots, it would already mean mixed segment/reservation types of > > memory > > > > > allocation in general (like batch and streaming). > > > > > Then I think memory manager would need some input from tasks > because > > it > > > > is > > > > > hard for Flink to decide how much to cache and which pool to > allocate > > > > from > > > > > eventually. > > > > > As I understand, tasks do not really need this now. That is why I > > think > > > > > having memory manager per slot will give more security by isolating > > > slots > > > > > with relatively small effort. > > > > > If there is no preallocation/caching then creating the memory > manager > > > is > > > > > almost no-op, later we can reconsider it if needed. What do you > > think? > > > > > > > > > > Thanks, > > > > > Andrey > > > > > > > > > > On Sat, Sep 14, 2019 at 8:11 AM Xintong Song < > [hidden email]> > > > > > wrote: > > > > > > > > > > > @Andrey, > > > > > > > > > > > > If we speak in the condition of dynamic slot allocation, then > > DataSet > > > > > tasks > > > > > > should always request slots with unknown resource profiles. Then > > the > > > > > > allocated slots should have 1/n on-heap managed memory and 1/n > > > off-heap > > > > > > managed memory of the task executor, where n is the configured > > > > > > 'numberOfSlot'. Given that DataSet operators only allocate > segments > > > (no > > > > > > reservation), and does not care whether the segments are on-heap > or > > > > > > off-heap, I think it makes sense to wrapping both pools for > DataSet > > > > > > operators, with respect to the on-heap/off-heap quota of the > > > allocated > > > > > > slot. > > > > > > > > > > > > I'm not sure whether we want to create one memory manager per > > slot. I > > > > > think > > > > > > we can have limit the quota of each slot with one memory manager > > per > > > > task > > > > > > executor. Given the dynamic slot allocation that dynamically > create > > > and > > > > > > destroy slots, I think having one memory manager per task > executor > > > may > > > > > save > > > > > > the overhead for frequently creating and destroying memory > > managers, > > > > and > > > > > > provide the chance for reusing memory segments across slots. > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 11:47 PM Andrey Zagrebin < > > > [hidden email] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > True, there would be no regression if only one type of memory > is > > > > > > > configured. This can be a problem only for the old jobs running > > in > > > a > > > > > > newly > > > > > > > configured cluster. > > > > > > > > > > > > > > About the pool type precedence, in general, it should not > matter > > > for > > > > > the > > > > > > > users which type the segments have. > > > > > > > The first implementation can be just to pull from any pool, > e.g. > > > > empty > > > > > > one > > > > > > > pool firstly and then another or some other random pulling. > > > > > > > This might be a problem if we mix segment allocations and > > > reservation > > > > > of > > > > > > > memory chunks from the same memory manager. > > > > > > > The reservation will be usually for a certain type of memory > then > > > the > > > > > > task > > > > > > > will probably have to also decide from which pool to allocate > the > > > > > > segments. > > > > > > > I would suggest we create a memory manager per slot and give it > > > > memory > > > > > > > limit of the slot then we do not have this kind of mixed > > operation > > > > > > > because Dataset/Batch jobs need only segment memory allocations > > and > > > > > > > streaming jobs need only memory chunks for state backends as I > > > > > understand > > > > > > > the current plan. > > > > > > > I would suggest we will look at it if we have the mixed > > operations > > > at > > > > > > some > > > > > > > point and it becomes a problem. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrey > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 5:24 PM Andrey Zagrebin < > > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > > > From: Xintong Song <[hidden email]> > > > > > > > > Date: Thu, Sep 12, 2019 at 4:21 AM > > > > > > > > Subject: Re: [DISCUSS] FLIP-49: Unified Memory Configuration > > for > > > > > > > > TaskExecutors > > > > > > > > To: dev <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > > > > > > > Thanks for bringing this up. > > > > > > > > > > > > > > > > If I understand correctly, this issue only occurs where the > > > cluster > > > > > is > > > > > > > > configured with both on-heap and off-heap memory. There > should > > be > > > > no > > > > > > > > regression for clusters configured in the old way (either all > > > > on-heap > > > > > > or > > > > > > > > all off-heap). > > > > > > > > > > > > > > > > I also agree that it would be good if the DataSet API jobs > can > > > use > > > > > both > > > > > > > > memory types. The only question I can see is that, from which > > > pool > > > > > > (heap > > > > > > > / > > > > > > > > off-heap) should we allocate memory for DataSet API > operators? > > Do > > > > we > > > > > > > > always prioritize one pool over the other? Or do we always > > > > prioritize > > > > > > the > > > > > > > > pool with more available memory left? > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 8:15 PM Andrey Zagrebin < > > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > While looking more into the implementation details of Step > 4, > > > we > > > > > > > released > > > > > > > > > during some offline discussions with @Till > > > > > > > > > that there can be a performance degradation for the batch > > > DataSet > > > > > API > > > > > > > if > > > > > > > > we > > > > > > > > > simply continue to pull memory from the pool > > > > > > > > > according the legacy option taskmanager.memory.off-heap. > > > > > > > > > > > > > > > > > > The reason is that if the cluster is newly configured to > > > > statically > > > > > > > split > > > > > > > > > heap/off-heap (not like previously either heap or 0ff-heap) > > > > > > > > > then the batch DataSet API jobs will be able to use only > one > > > type > > > > > of > > > > > > > > > memory. Although it does not really matter where the memory > > > > > segments > > > > > > > come > > > > > > > > > from > > > > > > > > > and potentially batch jobs can use both. Also, currently > the > > > > > Dataset > > > > > > > API > > > > > > > > > does not result in absolute resource requirements and its > > batch > > > > > jobs > > > > > > > will > > > > > > > > > always get a default share of TM resources. > > > > > > > > > > > > > > > > > > The suggestion is that we let the batch tasks of Dataset > API > > > pull > > > > > > from > > > > > > > > both > > > > > > > > > pools according to their fair slot share of each memory > type. > > > > > > > > > For that we can have a special wrapping view of both pools > > > which > > > > > will > > > > > > > > pull > > > > > > > > > segments (can be randomly) according to the slot limits. > > > > > > > > > The view can wrap TM level memory pools and be given to the > > > Task. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:35 PM Xintong Song < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks for your comments, Andrey. > > > > > > > > > > > > > > > > > > > > - Regarding Task Off-Heap Memory, I think you're right > that > > > the > > > > > > user > > > > > > > > need > > > > > > > > > > to make sure that direct memory and native memory > together > > > used > > > > > by > > > > > > > the > > > > > > > > > user > > > > > > > > > > code (external libs) do not exceed the configured value. > As > > > far > > > > > as > > > > > > I > > > > > > > > can > > > > > > > > > > think of, there is nothing we can do about it. > > > > > > > > > > > > > > > > > > > > I addressed the rest of your comment in the wiki page > [1]. > > > > Please > > > > > > > take > > > > > > > > a > > > > > > > > > > look. > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 6:13 PM Andrey Zagrebin < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > EDIT: sorry for confusion I meant > > > > > > > > > > > taskmanager.memory.off-heap > > > > > > > > > > > instead of > > > > > > > > > > > setting taskmanager.memory.preallocate > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 11:29 AM Andrey Zagrebin < > > > > > > > > [hidden email]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > @Xitong thanks a lot for driving the discussion. > > > > > > > > > > > > > > > > > > > > > > > > I also reviewed the FLIP and it looks quite good to > me. > > > > > > > > > > > > Here are some comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I wanted to discuss is the > > > > > > backwards-compatibility > > > > > > > > > with > > > > > > > > > > > > the previous user setups. We could list which > > options > > > we > > > > > > plan > > > > > > > to > > > > > > > > > > > deprecate. > > > > > > > > > > > > From the first glance it looks possible to provide > > the > > > > > > > > > same/similar > > > > > > > > > > > > behaviour for the setups relying on the deprecated > > > > > options. > > > > > > > E.g. > > > > > > > > > > > > setting taskmanager.memory.preallocate to true > could > > > > > > override > > > > > > > > the > > > > > > > > > > > > new taskmanager.memory.managed.offheap-fraction > to 1 > > > > etc. > > > > > At > > > > > > > the > > > > > > > > > > > moment the > > > > > > > > > > > > FLIP just states that in some cases it may require > > > > > > > > re-configuring > > > > > > > > > of > > > > > > > > > > > > cluster if migrated from prior versions. My > > suggestion > > > > is > > > > > > that > > > > > > > > we > > > > > > > > > > try > > > > > > > > > > > to > > > > > > > > > > > > keep it backwards-compatible unless there is a > good > > > > reason > > > > > > > like > > > > > > > > > some > > > > > > > > > > > major > > > > > > > > > > > > complication for the implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also couple of smaller things: > > > > > > > > > > > > > > > > > > > > > > > > - I suggest we remove TaskExecutorSpecifics from > the > > > > FLIP > > > > > > and > > > > > > > > > leave > > > > > > > > > > > > some general wording atm, like 'data structure to > > > store' > > > > > or > > > > > > > > > 'utility > > > > > > > > > > > > classes'. When the classes are implemented, we put > > the > > > > > > > concrete > > > > > > > > > > class > > > > > > > > > > > > names. This way we can avoid confusion and stale > > > > > documents. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - As I understand, if user task uses native memory > > > (not > > > > > > direct > > > > > > > > > > memory, > > > > > > > > > > > > but e.g. unsafe.allocate or from external lib), > > there > > > > will > > > > > > be > > > > > > > no > > > > > > > > > > > > explicit guard against exceeding 'task off heap > > > memory'. > > > > > > Then > > > > > > > > user > > > > > > > > > > > should > > > > > > > > > > > > still explicitly make sure that her/his direct > > buffer > > > > > > > allocation > > > > > > > > > > plus > > > > > > > > > > > any > > > > > > > > > > > > other memory usages does not exceed value > announced > > as > > > > > 'task > > > > > > > off > > > > > > > > > > > heap'. I > > > > > > > > > > > > guess there is no so much that can be done about > it > > > > except > > > > > > > > > > mentioning > > > > > > > > > > > in > > > > > > > > > > > > docs, similar to controlling the heap state > backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 10:07 AM Yang Wang < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> I also agree that all the configuration should be > > > > calculated > > > > > > out > > > > > > > > of > > > > > > > > > > > >> TaskManager. > > > > > > > > > > > >> > > > > > > > > > > > >> So a full configuration should be generated before > > > > > TaskManager > > > > > > > > > > started. > > > > > > > > > > > >> > > > > > > > > > > > >> Override the calculated configurations through -D > now > > > > seems > > > > > > > > better. > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> Best, > > > > > > > > > > > >> > > > > > > > > > > > >> Yang > > > > > > > > > > > >> > > > > > > > > > > > >> Xintong Song <[hidden email]> 于2019年9月2日周一 > > > > > 上午11:39写道： > > > > > > > > > > > >> > > > > > > > > > > > >> > I just updated the FLIP wiki page [1], with the > > > > following > > > > > > > > changes: > > > > > > > > > > > >> > > > > > > > > > > > > >> > - Network memory uses JVM direct memory, and is > > > > > accounted > > > > > > > > when > > > > > > > > > > > >> setting > > > > > > > > > > > >> > JVM max direct memory size parameter. > > > > > > > > > > > >> > - Use dynamic configurations (`-Dkey=value`) to > > > pass > > > > > > > > calculated > > > > > > > > > > > >> memory > > > > > > > > > > > >> > configs into TaskExecutors, instead of ENV > > > variables. > > > > > > > > > > > >> > - Remove 'supporting memory reservation' from > the > > > > scope > > > > > > of > > > > > > > > this > > > > > > > > > > > FLIP. > > > > > > > > > > > >> > > > > > > > > > > > > >> > @till @stephan, please take another look see if > > there > > > > are > > > > > > any > > > > > > > > > other > > > > > > > > > > > >> > concerns. > > > > > > > > > > > >> > > > > > > > > > > > > >> > Thank you~ > > > > > > > > > > > >> > > > > > > > > > > > > >> > Xintong Song > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > [1] > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > >> > > > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song < > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > >> > > Sorry for the late response. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > - Regarding the `TaskExecutorSpecifics` naming, > > > let's > > > > > > > discuss > > > > > > > > > the > > > > > > > > > > > >> detail > > > > > > > > > > > >> > > in PR. > > > > > > > > > > > >> > > - Regarding passing parameters into the > > > > `TaskExecutor`, > > > > > +1 > > > > > > > for > > > > > > > > > > using > > > > > > > > > > > >> > > dynamic configuration at the moment, given that > > > there > > > > > are > > > > > > > more > > > > > > > > > > > >> questions > > > > > > > > > > > >> > to > > > > > > > > > > > >> > > be discussed to have a general framework for > > > > overwriting > > > > > > > > > > > >> configurations > > > > > > > > > > > >> > > with ENV variables. > > > > > > > > > > > >> > > - Regarding memory reservation, I double checked > > > with > > > > Yu > > > > > > and > > > > > > > > he > > > > > > > > > > will > > > > > > > > > > > >> take > > > > > > > > > > > >> > > care of it. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Thank you~ > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Xintong Song > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann < > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > >> What I forgot to add is that we could tackle > > > > specifying > > > > > > the > > > > > > > > > > > >> > configuration > > > > > > > > > > > >> > >> fully in an incremental way and that the full > > > > > > specification > > > > > > > > > > should > > > > > > > > > > > be > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> desired end state. > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till Rohrmann < > > > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > I think our goal should be that the > > configuration > > > > is > > > > > > > fully > > > > > > > > > > > >> specified > > > > > > > > > > > >> > >> when > > > > > > > > > > > >> > >> > the process is started. By considering the > > > internal > > > > > > > > > calculation > > > > > > > > > > > >> step > > > > > > > > > > > >> > to > > > > > > > > > > > >> > >> be > > > > > > > > > > > >> > >> > rather validate existing values and calculate > > > > missing > > > > > > > ones, > > > > > > > > > > these > > > > > > > > > > > >> two > > > > > > > > > > > >> > >> > proposal shouldn't even conflict (given > > > > determinism). > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > Since we don't want to change an existing > > > > > > > flink-conf.yaml, > > > > > > > > > > > >> specifying > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> > full configuration would require to pass in > the > > > > > options > > > > > > > > > > > >> differently. > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > One way could be the ENV variables approach. > > The > > > > > reason > > > > > > > why > > > > > > > > > I'm > > > > > > > > > > > >> trying > > > > > > > > > > > >> > >> to > > > > > > > > > > > >> > >> > exclude this feature from the FLIP is that I > > > > believe > > > > > it > > > > > > > > > needs a > > > > > > > > > > > bit > > > > > > > > > > > >> > more > > > > > > > > > > > >> > >> > discussion. Just some questions which come to > > my > > > > > mind: > > > > > > > What > > > > > > > > > > would > > > > > > > > > > > >> be > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> > exact format (FLINK_KEY_NAME)? Would we > > support a > > > > dot > > > > > > > > > separator > > > > > > > > > > > >> which > > > > > > > > > > > >> > is > > > > > > > > > > > >> > >> > supported by some systems (FLINK.KEY.NAME)? > If > > > we > > > > > > accept > > > > > > > > the > > > > > > > > > > dot > > > > > > > > > > > >> > >> > separator what would be the order of > precedence > > > if > > > > > > there > > > > > > > > are > > > > > > > > > > two > > > > > > > > > > > >> ENV > > > > > > > > > > > >> > >> > variables defined (FLINK_KEY_NAME and > > > > FLINK.KEY.NAME > > > > > )? > > > > > > > > What > > > > > > > > > is > > > > > > > > > > > the > > > > > > > > > > > >> > >> > precedence of env variable vs. dynamic > > > > configuration > > > > > > > value > > > > > > > > > > > >> specified > > > > > > > > > > > >> > >> via -D? > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > Another approach could be to pass in the > > dynamic > > > > > > > > > configuration > > > > > > > > > > > >> values > > > > > > > > > > > >> > >> via > > > > > > > > > > > >> > >> > `-Dkey=value` to the Flink process. For that > we > > > > don't > > > > > > > have > > > > > > > > to > > > > > > > > > > > >> change > > > > > > > > > > > >> > >> > anything because the functionality already > > > exists. > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > Cheers, > > > > > > > > > > > >> > >> > Till > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan > Ewen < > > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > wrote: > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> >> I see. Under the assumption of strict > > > determinism > > > > > that > > > > > > > > > should > > > > > > > > > > > >> work. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> The original proposal had this point "don't > > > > compute > > > > > > > inside > > > > > > > > > the > > > > > > > > > > > TM, > > > > > > > > > > > >> > >> compute > > > > > > > > > > > >> > >> >> outside and supply a full config", because > > that > > > > > > sounded > > > > > > > > more > > > > > > > > > > > >> > intuitive. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till > > Rohrmann < > > > > > > > > > > > >> [hidden email] > > > > > > > > > > > >> > > > > > > > > > > > > > >> > >> >> wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> >> > My understanding was that before starting > > the > > > > > Flink > > > > > > > > > process > > > > > > > > > > we > > > > > > > > > > > >> > call a > > > > > > > > > > > >> > >> >> > utility which calculates these values. I > > > assume > > > > > that > > > > > > > > this > > > > > > > > > > > >> utility > > > > > > > > > > > >> > >> will > > > > > > > > > > > >> > >> >> do > > > > > > > > > > > >> > >> >> > the calculation based on a set of > configured > > > > > values > > > > > > > > > (process > > > > > > > > > > > >> > memory, > > > > > > > > > > > >> > >> >> flink > > > > > > > > > > > >> > >> >> > memory, network memory etc.). Assuming > that > > > > these > > > > > > > values > > > > > > > > > > don't > > > > > > > > > > > >> > differ > > > > > > > > > > > >> > >> >> from > > > > > > > > > > > >> > >> >> > the values with which the JVM is started, > it > > > > > should > > > > > > be > > > > > > > > > > > possible > > > > > > > > > > > >> to > > > > > > > > > > > >> > >> >> > recompute them in the Flink process in > order > > > to > > > > > set > > > > > > > the > > > > > > > > > > > values. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan > > Ewen > > > < > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > When computing the values in the JVM > > process > > > > > after > > > > > > > it > > > > > > > > > > > started, > > > > > > > > > > > >> > how > > > > > > > > > > > >> > >> >> would > > > > > > > > > > > >> > >> >> > > you deal with values like Max Direct > > Memory, > > > > > > > Metaspace > > > > > > > > > > size. > > > > > > > > > > > >> > native > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > reservation (reduce heap size), etc? All > > the > > > > > > values > > > > > > > > that > > > > > > > > > > are > > > > > > > > > > > >> > >> >> parameters > > > > > > > > > > > >> > >> >> > to > > > > > > > > > > > >> > >> >> > > the JVM process and that need to be > > supplied > > > > at > > > > > > > > process > > > > > > > > > > > >> startup? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till > > > Rohrmann > > > > < > > > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > Thanks for the clarification. I have > > some > > > > more > > > > > > > > > comments: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > - I would actually split the logic to > > > > compute > > > > > > the > > > > > > > > > > process > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > > requirements and storing the values > into > > > two > > > > > > > things. > > > > > > > > > > E.g. > > > > > > > > > > > >> one > > > > > > > > > > > >> > >> could > > > > > > > > > > > >> > >> >> > name > > > > > > > > > > > >> > >> >> > > > the former TaskExecutorProcessUtility > > and > > > > the > > > > > > > > latter > > > > > > > > > > > >> > >> >> > > > TaskExecutorProcessMemory. But we can > > > > discuss > > > > > > this > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > > PR > > > > > > > > > > > >> > >> since > > > > > > > > > > > >> > >> >> it's > > > > > > > > > > > >> > >> >> > > > just a naming detail. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > - Generally, I'm not opposed to making > > > > > > > configuration > > > > > > > > > > > values > > > > > > > > > > > >> > >> >> overridable > > > > > > > > > > > >> > >> >> > > by > > > > > > > > > > > >> > >> >> > > > ENV variables. I think this is a very > > good > > > > > idea > > > > > > > and > > > > > > > > > > makes > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> >> > > > configurability of Flink processes > > easier. > > > > > > > However, > > > > > > > > I > > > > > > > > > > > think > > > > > > > > > > > >> > that > > > > > > > > > > > >> > >> >> adding > > > > > > > > > > > >> > >> >> > > > this functionality should not be part > of > > > > this > > > > > > FLIP > > > > > > > > > > because > > > > > > > > > > > >> it > > > > > > > > > > > >> > >> would > > > > > > > > > > > >> > >> >> > > simply > > > > > > > > > > > >> > >> >> > > > widen the scope unnecessarily. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > The reasons why I believe it is > > > unnecessary > > > > > are > > > > > > > the > > > > > > > > > > > >> following: > > > > > > > > > > > >> > >> For > > > > > > > > > > > >> > >> >> Yarn > > > > > > > > > > > >> > >> >> > > we > > > > > > > > > > > >> > >> >> > > > already create write a flink-conf.yaml > > > which > > > > > > could > > > > > > > > be > > > > > > > > > > > >> populated > > > > > > > > > > > >> > >> with > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > memory settings. For the other > processes > > > it > > > > > > should > > > > > > > > not > > > > > > > > > > > make > > > > > > > > > > > >> a > > > > > > > > > > > >> > >> >> > difference > > > > > > > > > > > >> > >> >> > > > whether the loaded Configuration is > > > > populated > > > > > > with > > > > > > > > the > > > > > > > > > > > >> memory > > > > > > > > > > > >> > >> >> settings > > > > > > > > > > > >> > >> >> > > from > > > > > > > > > > > >> > >> >> > > > ENV variables or by using > > > > > > > TaskExecutorProcessUtility > > > > > > > > > to > > > > > > > > > > > >> compute > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > missing > > > > > > > > > > > >> > >> >> > > > values from the loaded configuration. > If > > > the > > > > > > > latter > > > > > > > > > > would > > > > > > > > > > > >> not > > > > > > > > > > > >> > be > > > > > > > > > > > >> > >> >> > possible > > > > > > > > > > > >> > >> >> > > > (wrong or missing configuration > values), > > > > then > > > > > we > > > > > > > > > should > > > > > > > > > > > not > > > > > > > > > > > >> > have > > > > > > > > > > > >> > >> >> been > > > > > > > > > > > >> > >> >> > > able > > > > > > > > > > > >> > >> >> > > > to actually start the process in the > > first > > > > > > place. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > - Concerning the memory reservation: I > > > agree > > > > > > with > > > > > > > > you > > > > > > > > > > that > > > > > > > > > > > >> we > > > > > > > > > > > >> > >> need > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > memory reservation functionality to > make > > > > > > streaming > > > > > > > > > jobs > > > > > > > > > > > work > > > > > > > > > > > >> > with > > > > > > > > > > > >> > >> >> > > "managed" > > > > > > > > > > > >> > >> >> > > > memory. However, w/o this > functionality > > > the > > > > > > whole > > > > > > > > Flip > > > > > > > > > > > would > > > > > > > > > > > >> > >> already > > > > > > > > > > > >> > >> >> > > bring > > > > > > > > > > > >> > >> >> > > > a good amount of improvements to our > > users > > > > > when > > > > > > > > > running > > > > > > > > > > > >> batch > > > > > > > > > > > >> > >> jobs. > > > > > > > > > > > >> > >> >> > > > Moreover, by keeping the scope smaller > > we > > > > can > > > > > > > > complete > > > > > > > > > > the > > > > > > > > > > > >> FLIP > > > > > > > > > > > >> > >> >> faster. > > > > > > > > > > > >> > >> >> > > > Hence, I would propose to address the > > > memory > > > > > > > > > reservation > > > > > > > > > > > >> > >> >> functionality > > > > > > > > > > > >> > >> >> > > as a > > > > > > > > > > > >> > >> >> > > > follow up FLIP (which Yu is working on > > if > > > > I'm > > > > > > not > > > > > > > > > > > mistaken). > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > Cheers, > > > > > > > > > > > >> > >> >> > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM Yang > > > Wang < > > > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > Just add my 2 cents. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > Using environment variables to > > override > > > > the > > > > > > > > > > > configuration > > > > > > > > > > > >> for > > > > > > > > > > > >> > >> >> > different > > > > > > > > > > > >> > >> >> > > > > taskmanagers is better. > > > > > > > > > > > >> > >> >> > > > > We do not need to generate dedicated > > > > > > > > flink-conf.yaml > > > > > > > > > > for > > > > > > > > > > > >> all > > > > > > > > > > > >> > >> >> > > > taskmanagers. > > > > > > > > > > > >> > >> >> > > > > A common flink-conf.yam and > different > > > > > > > environment > > > > > > > > > > > >> variables > > > > > > > > > > > >> > are > > > > > > > > > > > >> > >> >> > enough. > > > > > > > > > > > >> > >> >> > > > > By reducing the distributed cached > > > files, > > > > it > > > > > > > could > > > > > > > > > > make > > > > > > > > > > > >> > >> launching > > > > > > > > > > > >> > >> >> a > > > > > > > > > > > >> > >> >> > > > > taskmanager faster. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > Stephan gives a good suggestion that > > we > > > > > could > > > > > > > move > > > > > > > > > the > > > > > > > > > > > >> logic > > > > > > > > > > > >> > >> into > > > > > > > > > > > >> > >> >> > > > > "GlobalConfiguration.loadConfig()" > > > method. > > > > > > > > > > > >> > >> >> > > > > Maybe the client could also benefit > > from > > > > > this. > > > > > > > > > > Different > > > > > > > > > > > >> > users > > > > > > > > > > > >> > >> do > > > > > > > > > > > >> > >> >> not > > > > > > > > > > > >> > >> >> > > > have > > > > > > > > > > > >> > >> >> > > > > to export FLINK_CONF_DIR to update > few > > > > > config > > > > > > > > > options. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > Best, > > > > > > > > > > > >> > >> >> > > > > Yang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > Stephan Ewen <[hidden email]> > > > > > 于2019年8月28日周三 > > > > > > > > > > 上午1:21写道： > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > One note on the Environment > > Variables > > > > and > > > > > > > > > > > Configuration > > > > > > > > > > > >> > >> >> discussion. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > My understanding is that passed > ENV > > > > > > variables > > > > > > > > are > > > > > > > > > > > added > > > > > > > > > > > >> to > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > > > > configuration in the > > > > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > > > >> > >> method > > > > > > > > > > > >> > >> >> (or > > > > > > > > > > > >> > >> >> > > > > > similar). > > > > > > > > > > > >> > >> >> > > > > > For all the code inside Flink, it > > > looks > > > > > like > > > > > > > the > > > > > > > > > > data > > > > > > > > > > > >> was > > > > > > > > > > > >> > in > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > config > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > start with, just that the scripts > > that > > > > > > compute > > > > > > > > the > > > > > > > > > > > >> > variables > > > > > > > > > > > >> > >> can > > > > > > > > > > > >> > >> >> > pass > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > values to the process without > > actually > > > > > > needing > > > > > > > > to > > > > > > > > > > > write > > > > > > > > > > > >> a > > > > > > > > > > > >> > >> file. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > For example the > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > > > >> method > > > > > > > > > > > >> > >> would > > > > > > > > > > > >> > >> >> > take > > > > > > > > > > > >> > >> >> > > > any > > > > > > > > > > > >> > >> >> > > > > > ENV variable prefixed with "flink" > > and > > > > add > > > > > > it > > > > > > > > as a > > > > > > > > > > > >> config > > > > > > > > > > > >> > >> key. > > > > > > > > > > > >> > >> >> > > > > > "flink_taskmanager_memory_size=2g" > > > would > > > > > > > become > > > > > > > > > > > >> > >> >> > > > "taskmanager.memory.size: > > > > > > > > > > > >> > >> >> > > > > > 2g". > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM > > > Xintong > > > > > > Song < > > > > > > > > > > > >> > >> >> > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thanks for the comments, Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > I've also seen your comments on > > the > > > > wiki > > > > > > > page, > > > > > > > > > but > > > > > > > > > > > >> let's > > > > > > > > > > > >> > >> keep > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > > > > discussion here. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding > > 'TaskExecutorSpecifics', > > > > how > > > > > > do > > > > > > > > you > > > > > > > > > > > think > > > > > > > > > > > >> > about > > > > > > > > > > > >> > >> >> > naming > > > > > > > > > > > >> > >> >> > > it > > > > > > > > > > > >> > >> >> > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > > > > > > >> > >> >> > > > > > > - Regarding passing memory > > > > > configurations > > > > > > > into > > > > > > > > > > task > > > > > > > > > > > >> > >> executors, > > > > > > > > > > > >> > >> >> > I'm > > > > > > > > > > > >> > >> >> > > in > > > > > > > > > > > >> > >> >> > > > > > favor > > > > > > > > > > > >> > >> >> > > > > > > of do it via environment > variables > > > > > rather > > > > > > > than > > > > > > > > > > > >> > >> configurations, > > > > > > > > > > > >> > >> >> > with > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > following two reasons. > > > > > > > > > > > >> > >> >> > > > > > > - It is easier to keep the > > memory > > > > > > options > > > > > > > > once > > > > > > > > > > > >> > calculate > > > > > > > > > > > >> > >> >> not to > > > > > > > > > > > >> > >> >> > > be > > > > > > > > > > > >> > >> >> > > > > > > changed with environment > variables > > > > > rather > > > > > > > than > > > > > > > > > > > >> > >> configurations. > > > > > > > > > > > >> > >> >> > > > > > > - I'm not sure whether we > should > > > > write > > > > > > the > > > > > > > > > > > >> > configuration > > > > > > > > > > > >> > >> in > > > > > > > > > > > >> > >> >> > > startup > > > > > > > > > > > >> > >> >> > > > > > > scripts. Writing changes into > the > > > > > > > > configuration > > > > > > > > > > > files > > > > > > > > > > > >> > when > > > > > > > > > > > >> > >> >> > running > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > startup scripts does not sounds > > > right > > > > to > > > > > > me. > > > > > > > > Or > > > > > > > > > we > > > > > > > > > > > >> could > > > > > > > > > > > >> > >> make > > > > > > > > > > > >> > >> >> a > > > > > > > > > > > >> > >> >> > > copy > > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > > >> > >> >> > > > > > > configuration files per flink > > > cluster, > > > > > and > > > > > > > > make > > > > > > > > > > the > > > > > > > > > > > >> task > > > > > > > > > > > >> > >> >> executor > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > load > > > > > > > > > > > >> > >> >> > > > > > > from the copy, and clean up the > > copy > > > > > after > > > > > > > the > > > > > > > > > > > >> cluster is > > > > > > > > > > > >> > >> >> > shutdown, > > > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > > > >> > >> >> > > > > > > is complicated. (I think this is > > > also > > > > > what > > > > > > > > > Stephan > > > > > > > > > > > >> means > > > > > > > > > > > >> > in > > > > > > > > > > > >> > >> >> his > > > > > > > > > > > >> > >> >> > > > comment > > > > > > > > > > > >> > >> >> > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > the wiki page?) > > > > > > > > > > > >> > >> >> > > > > > > - Regarding reserving memory, I > > > think > > > > > this > > > > > > > > > change > > > > > > > > > > > >> should > > > > > > > > > > > >> > be > > > > > > > > > > > >> > >> >> > > included > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > >> > >> >> > > > > > > this FLIP. I think a big part of > > > > > > motivations > > > > > > > > of > > > > > > > > > > this > > > > > > > > > > > >> FLIP > > > > > > > > > > > >> > >> is > > > > > > > > > > > >> > >> >> to > > > > > > > > > > > >> > >> >> > > unify > > > > > > > > > > > >> > >> >> > > > > > > memory configuration for > > streaming / > > > > > batch > > > > > > > and > > > > > > > > > > make > > > > > > > > > > > it > > > > > > > > > > > >> > easy > > > > > > > > > > > >> > >> >> for > > > > > > > > > > > >> > >> >> > > > > > configuring > > > > > > > > > > > >> > >> >> > > > > > > rocksdb memory. If we don't > > support > > > > > memory > > > > > > > > > > > >> reservation, > > > > > > > > > > > >> > >> then > > > > > > > > > > > >> > >> >> > > > streaming > > > > > > > > > > > >> > >> >> > > > > > jobs > > > > > > > > > > > >> > >> >> > > > > > > cannot use managed memory > (neither > > > > > on-heap > > > > > > > or > > > > > > > > > > > >> off-heap), > > > > > > > > > > > >> > >> which > > > > > > > > > > > >> > >> >> > > makes > > > > > > > > > > > >> > >> >> > > > > this > > > > > > > > > > > >> > >> >> > > > > > > FLIP incomplete. > > > > > > > > > > > >> > >> >> > > > > > > - Regarding network memory, I > > think > > > > you > > > > > > are > > > > > > > > > > right. I > > > > > > > > > > > >> > think > > > > > > > > > > > >> > >> we > > > > > > > > > > > >> > >> >> > > > probably > > > > > > > > > > > >> > >> >> > > > > > > don't need to change network > stack > > > > from > > > > > > > using > > > > > > > > > > direct > > > > > > > > > > > >> > >> memory to > > > > > > > > > > > >> > >> >> > > using > > > > > > > > > > > >> > >> >> > > > > > unsafe > > > > > > > > > > > >> > >> >> > > > > > > native memory. Network memory > size > > > is > > > > > > > > > > deterministic, > > > > > > > > > > > >> > >> cannot be > > > > > > > > > > > >> > >> >> > > > reserved > > > > > > > > > > > >> > >> >> > > > > > as > > > > > > > > > > > >> > >> >> > > > > > > managed memory does, and cannot > be > > > > > > > overused. I > > > > > > > > > > think > > > > > > > > > > > >> it > > > > > > > > > > > >> > >> also > > > > > > > > > > > >> > >> >> > works > > > > > > > > > > > >> > >> >> > > if > > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > > >> > >> >> > > > > > > simply keep using direct memory > > for > > > > > > network > > > > > > > > and > > > > > > > > > > > >> include > > > > > > > > > > > >> > it > > > > > > > > > > > >> > >> in > > > > > > > > > > > >> > >> >> jvm > > > > > > > > > > > >> > >> >> > > max > > > > > > > > > > > >> > >> >> > > > > > > direct memory size. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 PM > > Till > > > > > > > Rohrmann > > > > > > > > < > > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Hi Xintong, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > thanks for addressing the > > comments > > > > and > > > > > > > > adding > > > > > > > > > a > > > > > > > > > > > more > > > > > > > > > > > >> > >> >> detailed > > > > > > > > > > > >> > >> >> > > > > > > > implementation plan. I have a > > > couple > > > > > of > > > > > > > > > comments > > > > > > > > > > > >> > >> concerning > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > > > > > implementation plan: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > - The name > > `TaskExecutorSpecifics` > > > > is > > > > > > not > > > > > > > > > really > > > > > > > > > > > >> > >> >> descriptive. > > > > > > > > > > > >> > >> >> > > > > Choosing > > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > different name could help > here. > > > > > > > > > > > >> > >> >> > > > > > > > - I'm not sure whether I would > > > pass > > > > > the > > > > > > > > memory > > > > > > > > > > > >> > >> >> configuration to > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > > > TaskExecutor via environment > > > > > variables. > > > > > > I > > > > > > > > > think > > > > > > > > > > it > > > > > > > > > > > >> > would > > > > > > > > > > > >> > >> be > > > > > > > > > > > >> > >> >> > > better > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > write > > > > > > > > > > > >> > >> >> > > > > > > > it into the configuration one > > uses > > > > to > > > > > > > start > > > > > > > > > the > > > > > > > > > > TM > > > > > > > > > > > >> > >> process. > > > > > > > > > > > >> > >> >> > > > > > > > - If possible, I would exclude > > the > > > > > > memory > > > > > > > > > > > >> reservation > > > > > > > > > > > >> > >> from > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > add this as part of a > dedicated > > > > FLIP. > > > > > > > > > > > >> > >> >> > > > > > > > - If possible, then I would > > > exclude > > > > > > > changes > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > >> > >> network > > > > > > > > > > > >> > >> >> > stack > > > > > > > > > > > >> > >> >> > > > from > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > FLIP. Maybe we can simply say > > that > > > > the > > > > > > > > direct > > > > > > > > > > > memory > > > > > > > > > > > >> > >> needed > > > > > > > > > > > >> > >> >> by > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > > network > > > > > > > > > > > >> > >> >> > > > > > > > stack is the framework direct > > > memory > > > > > > > > > > requirement. > > > > > > > > > > > >> > >> Changing > > > > > > > > > > > >> > >> >> how > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > is allocated can happen in a > > > second > > > > > > step. > > > > > > > > This > > > > > > > > > > > would > > > > > > > > > > > >> > keep > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > scope > > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > FLIP smaller. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 > PM > > > > > Xintong > > > > > > > > Song < > > > > > > > > > > > >> > >> >> > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Hi everyone, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > I just updated the FLIP > > document > > > > on > > > > > > wiki > > > > > > > > > [1], > > > > > > > > > > > with > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > > following > > > > > > > > > > > >> > >> >> > > > > > > changes. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > - Removed open question > > > > regarding > > > > > > > > > > > MemorySegment > > > > > > > > > > > >> > >> >> > allocation. > > > > > > > > > > > >> > >> >> > > As > > > > > > > > > > > >> > >> >> > > > > > > > > discussed, we exclude > this > > > > topic > > > > > > from > > > > > > > > the > > > > > > > > > > > >> scope of > > > > > > > > > > > >> > >> this > > > > > > > > > > > >> > >> >> > > FLIP. > > > > > > > > > > > >> > >> >> > > > > > > > > - Updated content about > JVM > > > > > direct > > > > > > > > memory > > > > > > > > > > > >> > parameter > > > > > > > > > > > >> > >> >> > > according > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > recent > > > > > > > > > > > >> > >> >> > > > > > > > > discussions, and moved > the > > > > other > > > > > > > > options > > > > > > > > > to > > > > > > > > > > > >> > >> "Rejected > > > > > > > > > > > >> > >> >> > > > > > Alternatives" > > > > > > > > > > > >> > >> >> > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > moment. > > > > > > > > > > > >> > >> >> > > > > > > > > - Added implementation > > steps. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > [1] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > On Mon, Aug 19, 2019 at 7:16 > > PM > > > > > > Stephan > > > > > > > > > Ewen < > > > > > > > > > > > >> > >> >> > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong: Concerning "wait > > for > > > > > > memory > > > > > > > > > users > > > > > > > > > > > >> before > > > > > > > > > > > >> > >> task > > > > > > > > > > > >> > >> >> > > dispose > > > > > > > > > > > >> > >> >> > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > release": I agree, that's > > how > > > it > > > > > > > should > > > > > > > > > be. > > > > > > > > > > > >> Let's > > > > > > > > > > > >> > >> try it > > > > > > > > > > > >> > >> >> > out. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong @Jingsong: > > > Concerning " > > > > > JVM > > > > > > > > does > > > > > > > > > > not > > > > > > > > > > > >> wait > > > > > > > > > > > >> > >> for > > > > > > > > > > > >> > >> >> GC > > > > > > > > > > > >> > >> >> > > when > > > > > > > > > > > >> > >> >> > > > > > > > allocating > > > > > > > > > > > >> > >> >> > > > > > > > > > direct memory buffer": > There > > > > seems > > > > > > to > > > > > > > be > > > > > > > > > > > pretty > > > > > > > > > > > >> > >> >> elaborate > > > > > > > > > > > >> > >> >> > > logic > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > free > > > > > > > > > > > >> > >> >> > > > > > > > > > buffers when allocating > new > > > > ones. > > > > > > See > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Till: Maybe. If we assume > > > that > > > > > the > > > > > > > JVM > > > > > > > > > > > default > > > > > > > > > > > >> > works > > > > > > > > > > > >> > >> >> (like > > > > > > > > > > > >> > >> >> > > > going > > > > > > > > > > > >> > >> >> > > > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > option 2 and not setting > > > > > > > > > > > >> "-XX:MaxDirectMemorySize" > > > > > > > > > > > >> > at > > > > > > > > > > > >> > >> >> all), > > > > > > > > > > > >> > >> >> > > > then > > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > > >> > >> >> > > > > > > > think > > > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > > should be okay to set > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > >> to > > > > > > > > > > > >> > >> >> > > > > > > > > > "off_heap_managed_memory + > > > > > > > > direct_memory" > > > > > > > > > > even > > > > > > > > > > > >> if > > > > > > > > > > > >> > we > > > > > > > > > > > >> > >> use > > > > > > > > > > > >> > >> >> > > > RocksDB. > > > > > > > > > > > >> > >> >> > > > > > > That > > > > > > > > > > > >> > >> >> > > > > > > > > is a > > > > > > > > > > > >> > >> >> > > > > > > > > > big if, though, I honestly > > > have > > > > no > > > > > > > idea > > > > > > > > :D > > > > > > > > > > > >> Would be > > > > > > > > > > > >> > >> >> good to > > > > > > > > > > > >> > >> >> > > > > > > understand > > > > > > > > > > > >> > >> >> > > > > > > > > > this, though, because this > > > would > > > > > > > affect > > > > > > > > > > option > > > > > > > > > > > >> (2) > > > > > > > > > > > >> > >> and > > > > > > > > > > > >> > >> >> > option > > > > > > > > > > > >> > >> >> > > > > > (1.2). > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at > 4:44 > > > PM > > > > > > > Xintong > > > > > > > > > > Song < > > > > > > > > > > > >> > >> >> > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thanks for the inputs, > > > > Jingsong. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Let me try to summarize > > your > > > > > > points. > > > > > > > > > > Please > > > > > > > > > > > >> > correct > > > > > > > > > > > >> > >> >> me if > > > > > > > > > > > >> > >> >> > > I'm > > > > > > > > > > > >> > >> >> > > > > > > wrong. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - Memory consumers > > should > > > > > > always > > > > > > > > > avoid > > > > > > > > > > > >> > returning > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > > > > segments > > > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > memory manager while > > > there > > > > > are > > > > > > > > still > > > > > > > > > > > >> > un-cleaned > > > > > > > > > > > >> > >> >> > > > structures / > > > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > may use the memory. > > > > > Otherwise, > > > > > > it > > > > > > > > > would > > > > > > > > > > > >> cause > > > > > > > > > > > >> > >> >> serious > > > > > > > > > > > >> > >> >> > > > > problems > > > > > > > > > > > >> > >> >> > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > > >> > >> >> > > > > > > > > > > multiple consumers > > trying > > > > to > > > > > > use > > > > > > > > the > > > > > > > > > > same > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > segment. > > > > > > > > > > > >> > >> >> > > > > > > > > > > - JVM does not wait > for > > > GC > > > > > when > > > > > > > > > > > allocating > > > > > > > > > > > >> > >> direct > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > > > > buffer. > > > > > > > > > > > >> > >> >> > > > > > > > > > > Therefore even we set > > > > proper > > > > > > max > > > > > > > > > direct > > > > > > > > > > > >> memory > > > > > > > > > > > >> > >> size > > > > > > > > > > > >> > >> >> > > limit, > > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > > >> > >> >> > > > > > > may > > > > > > > > > > > >> > >> >> > > > > > > > > > still > > > > > > > > > > > >> > >> >> > > > > > > > > > > encounter direct > memory > > > oom > > > > > if > > > > > > > the > > > > > > > > GC > > > > > > > > > > > >> cleaning > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > >> > >> >> > > > slower > > > > > > > > > > > >> > >> >> > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct memory > > allocation. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Am I understanding this > > > > > correctly? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 at > > 4:21 > > > > PM > > > > > > > > > > JingsongLee > > > > > > > > > > > < > > > > > > > > > > > >> > >> >> > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > .invalid> > > > > > > > > > > > >> > >> >> > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Hi stephan: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About option 2: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > if additional threads > > not > > > > > > cleanly > > > > > > > > shut > > > > > > > > > > > down > > > > > > > > > > > >> > >> before > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > >> > >> >> > can > > > > > > > > > > > >> > >> >> > > > > exit > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > task: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > In the current case of > > > > memory > > > > > > > reuse, > > > > > > > > > it > > > > > > > > > > > has > > > > > > > > > > > >> > >> freed up > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > > > > uses. If this memory > is > > > > used > > > > > by > > > > > > > > other > > > > > > > > > > > tasks > > > > > > > > > > > >> > and > > > > > > > > > > > >> > >> >> > > > asynchronous > > > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of exited task may > > still > > > be > > > > > > > > writing, > > > > > > > > > > > there > > > > > > > > > > > >> > will > > > > > > > > > > > >> > >> be > > > > > > > > > > > >> > >> >> > > > > concurrent > > > > > > > > > > > >> > >> >> > > > > > > > > security > > > > > > > > > > > >> > >> >> > > > > > > > > > > > problems, and even > lead > > > to > > > > > > errors > > > > > > > > in > > > > > > > > > > user > > > > > > > > > > > >> > >> computing > > > > > > > > > > > >> > >> >> > > > results. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > So I think this is a > > > serious > > > > > and > > > > > > > > > > > intolerable > > > > > > > > > > > >> > >> bug, No > > > > > > > > > > > >> > >> >> > > matter > > > > > > > > > > > >> > >> >> > > > > > what > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > option is, it should > be > > > > > > avoided. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About direct memory > > > cleaned > > > > by > > > > > > GC: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > I don't think it is a > > good > > > > > idea, > > > > > > > > I've > > > > > > > > > > > >> > >> encountered so > > > > > > > > > > > >> > >> >> > many > > > > > > > > > > > >> > >> >> > > > > > > > situations > > > > > > > > > > > >> > >> >> > > > > > > > > > > > that it's too late > for > > GC > > > > to > > > > > > > cause > > > > > > > > > > > >> > DirectMemory > > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > > >> > >> >> > > > Release > > > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > allocate DirectMemory > > > > depend > > > > > on > > > > > > > the > > > > > > > > > > type > > > > > > > > > > > of > > > > > > > > > > > >> > user > > > > > > > > > > > >> > >> >> job, > > > > > > > > > > > >> > >> >> > > > which > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > often beyond our > > control. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Best, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Jingsong Lee > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > From:Stephan Ewen < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Send > > Time:2019年8月19日(星期一) > > > > > 15:56 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > To:dev < > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Subject:Re: [DISCUSS] > > > > FLIP-49: > > > > > > > > Unified > > > > > > > > > > > >> Memory > > > > > > > > > > > >> > >> >> > > Configuration > > > > > > > > > > > >> > >> >> > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > TaskExecutors > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > My main concern with > > > option > > > > 2 > > > > > > > > > (manually > > > > > > > > > > > >> release > > > > > > > > > > > >> > >> >> memory) > > > > > > > > > > > >> > >> >> > > is > > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > segfaults > > > > > > > > > > > >> > >> >> > > > > > > > > > > > in the JVM send off > all > > > > sorts > > > > > of > > > > > > > > > alarms > > > > > > > > > > on > > > > > > > > > > > >> user > > > > > > > > > > > >> > >> >> ends. > > > > > > > > > > > >> > >> >> > So > > > > > > > > > > > >> > >> >> > > we > > > > > > > > > > > >> > >> >> > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > guarantee that this > > never > > > > > > happens. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > The trickyness is in > > tasks > > > > > that > > > > > > > uses > > > > > > > > > > data > > > > > > > > > > > >> > >> >> structures / > > > > > > > > > > > >> > >> >> > > > > > algorithms > > > > > > > > > > > >> > >> >> > > > > > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > additional threads, > like > > > > hash > > > > > > > table > > > > > > > > > > > >> spill/read > > > > > > > > > > > >> > >> and > > > > > > > > > > > >> > >> >> > > sorting > > > > > > > > > > > >> > >> >> > > > > > > threads. > > > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > > > > > > to ensure that these > > > cleanly > > > > > > shut > > > > > > > > down > > > > > > > > > > > >> before > > > > > > > > > > > >> > we > > > > > > > > > > > >> > >> can > > > > > > > > > > > >> > >> >> > exit > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > task. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > I am not sure that we > > have > > > > > that > > > > > > > > > > guaranteed > > > > > > > > > > > >> > >> already, > > > > > > > > > > > >> > >> >> > > that's > > > > > > > > > > > >> > >> >> > > > > why > > > > > > > > > > > >> > >> >> > > > > > > > option > > > > > > > > > > > >> > >> >> > > > > > > > > > 1.1 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > seemed simpler to me. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 > at > > > 3:42 > > > > > PM > > > > > > > > > Xintong > > > > > > > > > > > >> Song < > > > > > > > > > > > >> > >> >> > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thanks for the > > comments, > > > > > > > Stephan. > > > > > > > > > > > >> Summarized > > > > > > > > > > > >> > in > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > >> > >> >> > > way > > > > > > > > > > > >> > >> >> > > > > > really > > > > > > > > > > > >> > >> >> > > > > > > > > makes > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > things easier to > > > > understand. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'm in favor of > option > > > 2, > > > > at > > > > > > > least > > > > > > > > > for > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> moment. I > > > > > > > > > > > >> > >> >> > > > think > > > > > > > > > > > >> > >> >> > > > > it > > > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > difficult to keep it > > > > > segfault > > > > > > > safe > > > > > > > > > for > > > > > > > > > > > >> memory > > > > > > > > > > > >> > >> >> > manager, > > > > > > > > > > > >> > >> >> > > as > > > > > > > > > > > >> > >> >> > > > > > long > > > > > > > > > > > >> > >> >> > > > > > > as > > > > > > > > > > > >> > >> >> > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > always > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > de-allocate the > memory > > > > > segment > > > > > > > > when > > > > > > > > > it > > > > > > > > > > > is > > > > > > > > > > > >> > >> released > > > > > > > > > > > >> > >> >> > from > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > consumers. Only if > the > > > > > memory > > > > > > > > > consumer > > > > > > > > > > > >> > continue > > > > > > > > > > > >> > >> >> using > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > > buffer > > > > > > > > > > > >> > >> >> > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > segment after > > releasing > > > > it, > > > > > in > > > > > > > > which > > > > > > > > > > > case > > > > > > > > > > > >> we > > > > > > > > > > > >> > do > > > > > > > > > > > >> > >> >> want > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > job > > > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > fail > > > > > > > > > > > >> > >> >> > > > > > > > > > so > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > detect the memory > leak > > > > > early. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.2, I > > don't > > > > > think > > > > > > > this > > > > > > > > > is > > > > > > > > > > a > > > > > > > > > > > >> good > > > > > > > > > > > >> > >> idea. > > > > > > > > > > > >> > >> >> > Not > > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > > >> > >> >> > > > > > > > because > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > assumption (regular > GC > > > is > > > > > > enough > > > > > > > > to > > > > > > > > > > > clean > > > > > > > > > > > >> > >> direct > > > > > > > > > > > >> > >> >> > > buffers) > > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > always > > > > > > > > > > > >> > >> >> > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > true, but also it > > makes > > > > > harder > > > > > > > for > > > > > > > > > > > finding > > > > > > > > > > > >> > >> >> problems > > > > > > > > > > > >> > >> >> > in > > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > > >> > >> >> > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > overuse. E.g., user > > > > > configured > > > > > > > > some > > > > > > > > > > > direct > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> for > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > libraries. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > If the library > > actually > > > > use > > > > > > more > > > > > > > > > > direct > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> then > > > > > > > > > > > >> > >> >> > > > > > configured, > > > > > > > > > > > >> > >> >> > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cannot be cleaned by > > GC > > > > > > because > > > > > > > > they > > > > > > > > > > are > > > > > > > > > > > >> > still > > > > > > > > > > > >> > >> in > > > > > > > > > > > >> > >> >> > use, > > > > > > > > > > > >> > >> >> > > > may > > > > > > > > > > > >> > >> >> > > > > > lead > > > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > overuse > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > of the total > container > > > > > memory. > > > > > > > In > > > > > > > > > that > > > > > > > > > > > >> case, > > > > > > > > > > > >> > >> if it > > > > > > > > > > > >> > >> >> > > didn't > > > > > > > > > > > >> > >> >> > > > > > touch > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > default max direct > > > memory > > > > > > limit, > > > > > > > > we > > > > > > > > > > > cannot > > > > > > > > > > > >> > get > > > > > > > > > > > >> > >> a > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > OOM > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > will become super > hard > > > to > > > > > > > > understand > > > > > > > > > > > which > > > > > > > > > > > >> > >> part of > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to be updated. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.1, it > has > > > the > > > > > > > similar > > > > > > > > > > > >> problem as > > > > > > > > > > > >> > >> >> 1.2, if > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > > > exceeded > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > memory does not > reach > > > the > > > > > max > > > > > > > > direct > > > > > > > > > > > >> memory > > > > > > > > > > > >> > >> limit > > > > > > > > > > > >> > >> >> > > > specified > > > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > dedicated > parameter. I > > > > think > > > > > > it > > > > > > > is > > > > > > > > > > > >> slightly > > > > > > > > > > > >> > >> better > > > > > > > > > > > >> > >> >> > than > > > > > > > > > > > >> > >> >> > > > > 1.2, > > > > > > > > > > > >> > >> >> > > > > > > only > > > > > > > > > > > >> > >> >> > > > > > > > > > > because > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we can tune the > > > parameter. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Mon, Aug 19, 2019 > > at > > > > 2:53 > > > > > > PM > > > > > > > > > > Stephan > > > > > > > > > > > >> Ewen > > > > > > > > > > > >> > < > > > > > > > > > > > >> > >> >> > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > About the > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > >> > >> discussion, > > > > > > > > > > > >> > >> >> > maybe > > > > > > > > > > > >> > >> >> > > > let > > > > > > > > > > > >> > >> >> > > > > > me > > > > > > > > > > > >> > >> >> > > > > > > > > > > summarize > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > it a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > bit differently: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > We have the > > following > > > > two > > > > > > > > options: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (1) We let > > > > MemorySegments > > > > > be > > > > > > > > > > > >> de-allocated > > > > > > > > > > > >> > by > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > GC. > > > > > > > > > > > >> > >> >> > > > That > > > > > > > > > > > >> > >> >> > > > > > > makes > > > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > segfault > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > safe. But then we > > > need a > > > > > way > > > > > > > to > > > > > > > > > > > trigger > > > > > > > > > > > >> GC > > > > > > > > > > > >> > in > > > > > > > > > > > >> > >> >> case > > > > > > > > > > > >> > >> >> > > > > > > > de-allocation > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > re-allocation of a > > > bunch > > > > > of > > > > > > > > > segments > > > > > > > > > > > >> > happens > > > > > > > > > > > >> > >> >> > quickly, > > > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > often > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > case during batch > > > > > scheduling > > > > > > > or > > > > > > > > > task > > > > > > > > > > > >> > restart. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - The > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > >> (option > > > > > > > > > > > >> > >> 1.1) > > > > > > > > > > > >> > >> >> is > > > > > > > > > > > >> > >> >> > one > > > > > > > > > > > >> > >> >> > > > way > > > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Another way > > could > > > be > > > > > to > > > > > > > > have a > > > > > > > > > > > >> > dedicated > > > > > > > > > > > >> > >> >> > > > bookkeeping > > > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > MemoryManager > > (option > > > > > 1.2), > > > > > > so > > > > > > > > > that > > > > > > > > > > > this > > > > > > > > > > > >> > is a > > > > > > > > > > > >> > >> >> > number > > > > > > > > > > > >> > >> >> > > > > > > > independent > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > parameter. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (2) We manually > > > allocate > > > > > and > > > > > > > > > > > de-allocate > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > >> > >> >> > > for > > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > MemorySegments > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (option 2). That > way > > > we > > > > > need > > > > > > > not > > > > > > > > > > worry > > > > > > > > > > > >> > about > > > > > > > > > > > >> > >> >> > > triggering > > > > > > > > > > > >> > >> >> > > > > GC > > > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > some > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > threshold or > > > > bookkeeping, > > > > > > but > > > > > > > it > > > > > > > > > is > > > > > > > > > > > >> harder > > > > > > > > > > > >> > to > > > > > > > > > > > >> > >> >> > prevent > > > > > > > > > > > >> > >> >> > > > > > > > segfaults. > > > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be very careful > > about > > > > when > > > > > > we > > > > > > > > > > release > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > > > segments > > > > > > > > > > > >> > >> >> > > > > > > (only > > > > > > > > > > > >> > >> >> > > > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cleanup phase of > the > > > > main > > > > > > > > thread). > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with > option > > > > 1.1, > > > > > we > > > > > > > > > > probably > > > > > > > > > > > >> need > > > > > > > > > > > >> > to > > > > > > > > > > > >> > >> >> set > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > to > > > > > > > > > > > >> > >> >> > > "off_heap_managed_memory + > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct_memory" > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > have > "direct_memory" > > > as > > > > a > > > > > > > > separate > > > > > > > > > > > >> reserved > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > >> > >> >> > > > pool. > > > > > > > > > > > >> > >> >> > > > > > > > Because > > > > > > > > > > > >> > >> >> > > > > > > > > if > > > > > > > > > > > >> > >> >> > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > just > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > set > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > "off_heap_managed_memory + > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > jvm_overhead", > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > then there will be > > > times > > > > > > when > > > > > > > > that > > > > > > > > > > > >> entire > > > > > > > > > > > >> > >> >> memory is > > > > > > > > > > > >> > >> >> > > > > > allocated > > > > > > > > > > > >> > >> >> > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > buffers and we > have > > > > > nothing > > > > > > > left > > > > > > > > > for > > > > > > > > > > > the > > > > > > > > > > > >> > JVM > > > > > > > > > > > >> > >> >> > > overhead. > > > > > > > > > > > >> > >> >> > > > So > > > > > > > > > > > >> > >> >> > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > either > > > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > way to compensate > > for > > > > that > > > > > > > > (again > > > > > > > > > > some > > > > > > > > > > > >> > safety > > > > > > > > > > > >> > >> >> > margin > > > > > > > > > > > >> > >> >> > > > > cutoff > > > > > > > > > > > >> > >> >> > > > > > > > > value) > > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > will exceed > > container > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with > option > > > > 1.2, > > > > > we > > > > > > > > need > > > > > > > > > to > > > > > > > > > > > be > > > > > > > > > > > >> > aware > > > > > > > > > > > >> > >> >> that > > > > > > > > > > > >> > >> >> > it > > > > > > > > > > > >> > >> >> > > > > takes > > > > > > > > > > > >> > >> >> > > > > > > > > > elaborate > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > logic > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > to push recycling > of > > > > > direct > > > > > > > > > buffers > > > > > > > > > > > >> without > > > > > > > > > > > >> > >> >> always > > > > > > > > > > > >> > >> >> > > > > > > triggering a > > > > > > > > > > > >> > >> >> > > > > > > > > > full > > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My first guess is > > that > > > > the > > > > > > > > options > > > > > > > > > > > will > > > > > > > > > > > >> be > > > > > > > > > > > >> > >> >> easiest > > > > > > > > > > > >> > >> >> > to > > > > > > > > > > > >> > >> >> > > > do > > > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > following > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > order: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.1 > with > > a > > > > > > > dedicated > > > > > > > > > > > >> > direct_memory > > > > > > > > > > > >> > >> >> > > > parameter, > > > > > > > > > > > >> > >> >> > > > > as > > > > > > > > > > > >> > >> >> > > > > > > > > > discussed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > above. We would > need > > > to > > > > > > find a > > > > > > > > way > > > > > > > > > > to > > > > > > > > > > > >> set > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > > > > direct_memory > > > > > > > > > > > >> > >> >> > > > > > > > > > parameter > > > > > > > > > > > >> > >> >> > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > default. We could > > > start > > > > > with > > > > > > > 64 > > > > > > > > MB > > > > > > > > > > and > > > > > > > > > > > >> see > > > > > > > > > > > >> > >> how > > > > > > > > > > > >> > >> >> it > > > > > > > > > > > >> > >> >> > > goes > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > practice. > > > > > > > > > > > >> > >> >> > > > > > > > > > > One > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > danger I see is > that > > > > > setting > > > > > > > > this > > > > > > > > > > loo > > > > > > > > > > > >> low > > > > > > > > > > > >> > can > > > > > > > > > > > >> > >> >> > cause a > > > > > > > > > > > >> > >> >> > > > > bunch > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > additional > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GCs compared to > > before > > > > (we > > > > > > > need > > > > > > > > to > > > > > > > > > > > watch > > > > > > > > > > > >> > this > > > > > > > > > > > >> > >> >> > > > carefully). > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 2. It > is > > > > > actually > > > > > > > > quite > > > > > > > > > > > >> simple > > > > > > > > > > > >> > to > > > > > > > > > > > >> > >> >> > > implement, > > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > try > > > > > > > > > > > >> > >> >> > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > segfault safe we > are > > > at > > > > > the > > > > > > > > > moment. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.2: We > > > would > > > > > not > > > > > > > > touch > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > parameter > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > at all and assume > > that > > > > all > > > > > > the > > > > > > > > > > direct > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > > allocations > > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Netty do are > > > infrequent > > > > > > enough > > > > > > > > to > > > > > > > > > be > > > > > > > > > > > >> > cleaned > > > > > > > > > > > >> > >> up > > > > > > > > > > > >> > >> >> > fast > > > > > > > > > > > >> > >> >> > > > > enough > > > > > > > > > > > >> > >> >> > > > > > > > > through > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > regular > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GC. I am not sure > if > > > > that > > > > > > is a > > > > > > > > > valid > > > > > > > > > > > >> > >> assumption, > > > > > > > > > > > >> > >> >> > > > though. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Best, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Stephan > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > 2:16 > > > > > > > PM > > > > > > > > > > > Xintong > > > > > > > > > > > >> > Song > > > > > > > > > > > >> > >> < > > > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for > sharing > > > > your > > > > > > > > opinion > > > > > > > > > > > Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in > favor > > of > > > > > > > > alternative > > > > > > > > > > 2. > > > > > > > > > > > I > > > > > > > > > > > >> was > > > > > > > > > > > >> > >> >> > wondering > > > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > > > Unsafe.allocate() > > > > > > for > > > > > > > > > > off-heap > > > > > > > > > > > >> > >> managed > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > > and > > > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. > But > > > > after > > > > > > > > giving > > > > > > > > > > it a > > > > > > > > > > > >> > second > > > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > > > >> > >> >> > > > I > > > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 > > using > > > > > direct > > > > > > > > > memory > > > > > > > > > > > for > > > > > > > > > > > >> > >> off-heap > > > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > > > concern, > > > > > I > > > > > > > > think > > > > > > > > > > what > > > > > > > > > > > >> > >> proposed > > > > > > > > > > > >> > >> >> in > > > > > > > > > > > >> > >> >> > > this > > > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > > > > memory > > > > > > and > > > > > > > > > > network > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Unsafe.allocate(), > > > > which > > > > > > > means > > > > > > > > > > they > > > > > > > > > > > >> are > > > > > > > > > > > >> > >> >> > practically > > > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM > max > > > > > direct > > > > > > > > > memory. > > > > > > > > > > > The > > > > > > > > > > > >> > only > > > > > > > > > > > >> > >> >> parts > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct > memory > > > are > > > > > task > > > > > > > > > > off-heap > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> and > > > > > > > > > > > >> > >> >> > JVM > > > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly > > alternative > > > 2 > > > > > > > suggests > > > > > > > > > to > > > > > > > > > > > set > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> JVM > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > 1:48 > > > > > > > > PM > > > > > > > > > > Till > > > > > > > > > > > >> > >> Rohrmann > > > > > > > > > > > >> > >> >> < > > > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > > > > clarification > > > > > > > > > > > >> Xintong. I > > > > > > > > > > > >> > >> >> > > understand > > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in > > > favour > > > > > of > > > > > > > > > option 2 > > > > > > > > > > > >> > because > > > > > > > > > > > >> > >> it > > > > > > > > > > > >> > >> >> > makes > > > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit > the > > > > direct > > > > > > > > > memory, I > > > > > > > > > > > >> fear > > > > > > > > > > > >> > >> that > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > >> > >> >> > > might > > > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as > we > > > are > > > > > > > > currently > > > > > > > > > > in: > > > > > > > > > > > >> The > > > > > > > > > > > >> > >> user > > > > > > > > > > > >> > >> >> > might > > > > > > > > > > > >> > >> >> > > > see > > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the > OS > > > and > > > > > > does > > > > > > > > not > > > > > > > > > > know > > > > > > > > > > > >> why > > > > > > > > > > > >> > >> this > > > > > > > > > > > >> > >> >> is > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > decrease > > > > the > > > > > > > > process > > > > > > > > > > > memory > > > > > > > > > > > >> > size > > > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in > order > > to > > > > > > > > accommodate > > > > > > > > > > for > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> extra > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > decrease > > > > > memory > > > > > > > > > budgets > > > > > > > > > > > >> which > > > > > > > > > > > >> > >> are > > > > > > > > > > > >> > >> >> not > > > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the > > overall > > > > > > memory > > > > > > > > > > > >> consumption. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug > 16, > > > 2019 > > > > > at > > > > > > > > 11:01 > > > > > > > > > AM > > > > > > > > > > > >> > Xintong > > > > > > > > > > > >> > >> >> Song < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me > explain > > > > this > > > > > > > with a > > > > > > > > > > > >> concrete > > > > > > > > > > > >> > >> >> example > > > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we > > > have > > > > > the > > > > > > > > > > following > > > > > > > > > > > >> > >> scenario. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total > Process > > > > > Memory: > > > > > > > 1GB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > > > Memory > > > > > > (Task > > > > > > > > > > Off-Heap > > > > > > > > > > > >> > >> Memory + > > > > > > > > > > > >> > >> >> JVM > > > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory > > > (JVM > > > > > Heap > > > > > > > > > Memory, > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > Memory): > > > > > 800MB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > alternative > > > 2, > > > > > we > > > > > > > set > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > alternative > > > 3, > > > > > we > > > > > > > set > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say > 1TB. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > actual > > > > direct > > > > > > > > memory > > > > > > > > > > > usage > > > > > > > > > > > >> of > > > > > > > > > > > >> > >> Task > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not > exceed > > > > 200MB, > > > > > > > then > > > > > > > > > > > >> > alternative 2 > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same > utility. > > > > > Setting > > > > > > > > larger > > > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the > > > other > > > > > > > memory > > > > > > > > > > pools. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > actual > > > > direct > > > > > > > > memory > > > > > > > > > > > usage > > > > > > > > > > > >> of > > > > > > > > > > > >> > >> Task > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > > > potentially > > > > > > > > exceed > > > > > > > > > > > 200MB, > > > > > > > > > > > >> > then > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > > > Alternative 2 > > > > > > > suffers > > > > > > > > > > from > > > > > > > > > > > >> > >> frequent > > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > > >> > >> >> > > To > > > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can > do > > > is > > > > to > > > > > > > > modify > > > > > > > > > > the > > > > > > > > > > > >> > >> >> configuration > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > > > Off-Heap > > > > > > > Memory + > > > > > > > > > JVM > > > > > > > > > > > >> > >> Overhead). > > > > > > > > > > > >> > >> >> > Let's > > > > > > > > > > > >> > >> >> > > > say > > > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct > > Memory > > > > to > > > > > > > 250MB, > > > > > > > > > > this > > > > > > > > > > > >> will > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to > > > 750MB, > > > > > > given > > > > > > > > the > > > > > > > > > > > total > > > > > > > > > > > >> > >> process > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > > > alternative > > > > > > 3, > > > > > > > > > there > > > > > > > > > > is > > > > > > > > > > > >> no > > > > > > > > > > > >> > >> >> chance of > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of > > exceeding > > > > the > > > > > > > total > > > > > > > > > > > process > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > limit, > > > > > > > > > > > >> > >> >> > > > but > > > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use > up > > > all > > > > > the > > > > > > > > > reserved > > > > > > > > > > > >> native > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, > JVM > > > > > > > Metaspace), > > > > > > > > > if > > > > > > > > > > > the > > > > > > > > > > > >> > >> actual > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very > > > close > > > > to > > > > > > > > 200MB, > > > > > > > > > > user > > > > > > > > > > > >> > >> probably > > > > > > > > > > > >> > >> >> do > > > > > > > > > > > >> > >> >> > > not > > > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I > > > think > > > > > > from > > > > > > > > the > > > > > > > > > > > user's > > > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for > > alternative > > > 2 > > > > > may > > > > > > > lead > > > > > > > > > to > > > > > > > > > > > >> lower > > > > > > > > > > > >> > >> >> resource > > > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative > 3. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug > > 16, > > > > 2019 > > > > > > at > > > > > > > > > 10:28 > > > > > > > > > > AM > > > > > > > > > > > >> Till > > > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > > > >> > >> >> > > < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess > you > > > have > > > > > to > > > > > > > help > > > > > > > > > me > > > > > > > > > > > >> > >> understand > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative > 2 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt > to > > > > > memory > > > > > > > > under > > > > > > > > > > > >> > utilization > > > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > Alternative > > > 2: > > > > > set > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. > > Then > > > > > there > > > > > > > is > > > > > > > > > the > > > > > > > > > > > risk > > > > > > > > > > > >> > that > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > >> > >> >> > > size > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of > > garbage > > > > > > > > collection > > > > > > > > > > and > > > > > > > > > > > >> > >> >> potentially > > > > > > > > > > > >> > >> >> > an > > > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > Alternative > > > 3: > > > > > set > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > alternative > > 2. > > > > > This > > > > > > > > would > > > > > > > > > of > > > > > > > > > > > >> course > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > > > > alternative > > > > > > > 2 > > > > > > > > > now > > > > > > > > > > > >> result > > > > > > > > > > > >> > >> in an > > > > > > > > > > > >> > >> >> > > under > > > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared > to > > > > > > > alternative > > > > > > > > 3? > > > > > > > > > > If > > > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > memory > > > > size > > > > > > and > > > > > > > > we > > > > > > > > > > use > > > > > > > > > > > >> only > > > > > > > > > > > >> > >> >> little, > > > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > alternative > > 3 > > > > > > results > > > > > > > in > > > > > > > > > > > memory > > > > > > > > > > > >> > under > > > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, > Aug > > > 13, > > > > > 2019 > > > > > > > at > > > > > > > > > 4:19 > > > > > > > > > > > PM > > > > > > > > > > > >> > Yang > > > > > > > > > > > >> > >> >> Wang < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > > > xintong,till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native > > and > > > > > > Direct > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point > > is > > > > > > setting > > > > > > > a > > > > > > > > > very > > > > > > > > > > > >> large > > > > > > > > > > > >> > >> max > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > differentiate > > > > > > direct > > > > > > > > and > > > > > > > > > > > >> native > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > >> > >> >> > If > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory,including > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > > > memory > > > > > and > > > > > > > > > > framework > > > > > > > > > > > >> > direct > > > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in > > > favor > > > > of > > > > > > > > setting > > > > > > > > > > > >> direct > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > with > > > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > > > > Calculation > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree > > with > > > > > > > xintong. > > > > > > > > > For > > > > > > > > > > > Yarn > > > > > > > > > > > >> > and > > > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > > > >> > >> >> > > > need > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > configurations > > > > > in > > > > > > > > client > > > > > > > > > > to > > > > > > > > > > > >> avoid > > > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the > flink > > > > > master. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong > > > Song < > > > > > > > > > > > >> > >> [hidden email] > > > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > > > > replying, > > > > > > > > > > Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > > > > MemorySegment, I > > > > > > > > > > > think > > > > > > > > > > > >> > you > > > > > > > > > > > >> > >> are > > > > > > > > > > > >> > >> >> > > right > > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue > in > > > the > > > > > > scope > > > > > > > > of > > > > > > > > > > this > > > > > > > > > > > >> > FLIP. > > > > > > > > > > > >> > >> >> This > > > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > configure > > > > > memory > > > > > > > > pools > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > > > > consumers > > > > > > > use > > > > > > > > > it. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > direct > > > > > > > > memory, I > > > > > > > > > > > think > > > > > > > > > > > >> > >> >> > alternative > > > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > reservation > > > > > > issue > > > > > > > > that > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > >> > >> >> > > does, > > > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using > > > memory > > > > > at > > > > > > > the > > > > > > > > > > > >> container > > > > > > > > > > > >> > >> level, > > > > > > > > > > > >> > >> >> > > which > > > > > > > > > > > >> > >> >> > > > is > > > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both > > "Task > > > > > > > Off-Heap > > > > > > > > > > > Memory" > > > > > > > > > > > >> and > > > > > > > > > > > >> > >> "JVM > > > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > alternative > > > > 2, > > > > > > > users > > > > > > > > > > might > > > > > > > > > > > >> > >> configure > > > > > > > > > > > >> > >> >> > them > > > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just > to > > > > avoid > > > > > > > > getting > > > > > > > > > a > > > > > > > > > > > >> direct > > > > > > > > > > > >> > >> OOM. > > > > > > > > > > > >> > >> >> For > > > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > OOM, > > > > so > > > > > > > they > > > > > > > > > may > > > > > > > > > > > not > > > > > > > > > > > >> > >> config > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > two > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > > > consequences > > > > > > > are > > > > > > > > > > risks > > > > > > > > > > > >> of > > > > > > > > > > > >> > >> >> overall > > > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank > > you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Xintong > > > Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On > Tue, > > > Aug > > > > > 13, > > > > > > > 2019 > > > > > > > > > at > > > > > > > > > > > >> 9:39 AM > > > > > > > > > > > >> > >> Till > > > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > > > > proposing > > > > > > > > > > > this > > > > > > > > > > > >> > FLIP > > > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All > in > > > > all I > > > > > > > think > > > > > > > > > it > > > > > > > > > > > >> already > > > > > > > > > > > >> > >> >> looks > > > > > > > > > > > >> > >> >> > > quite > > > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > question > > > > > about > > > > > > > > > > > allocating > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > strictly > > > > > > > necessary > > > > > > > > > to > > > > > > > > > > do > > > > > > > > > > > >> in > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > context > > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > > done > > > > as a > > > > > > > > follow > > > > > > > > > > up? > > > > > > > > > > > >> > Without > > > > > > > > > > > >> > >> >> > knowing > > > > > > > > > > > >> > >> >> > > > all > > > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that > > we > > > > > would > > > > > > > > widen > > > > > > > > > > the > > > > > > > > > > > >> scope > > > > > > > > > > > >> > >> of > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to > > touch > > > > all > > > > > > the > > > > > > > > > > > existing > > > > > > > > > > > >> > call > > > > > > > > > > > >> > >> >> sites > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > segments > > > > > > > > > (this > > > > > > > > > > > >> should > > > > > > > > > > > >> > >> >> mainly > > > > > > > > > > > >> > >> >> > be > > > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > memory > > > > > > > > > reservation > > > > > > > > > > > >> call > > > > > > > > > > > >> > to > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this > > > and I > > > > > > would > > > > > > > > > hope > > > > > > > > > > > that > > > > > > > > > > > >> > >> this is > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > streaming > > > > > job > > > > > > > > would > > > > > > > > > > have > > > > > > > > > > > >> with > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Concerning > > > > > the > > > > > > > > > second > > > > > > > > > > > open > > > > > > > > > > > >> > >> >> question > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > direct > > > > > memory > > > > > > > > > limit, I > > > > > > > > > > > >> would > > > > > > > > > > > >> > >> also > > > > > > > > > > > >> > >> >> be > > > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > leaving > > > it > > > > > > open > > > > > > > > > would > > > > > > > > > > be > > > > > > > > > > > >> > best. > > > > > > > > > > > >> > >> My > > > > > > > > > > > >> > >> >> > > concern > > > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > in > > a > > > > > > similar > > > > > > > > > > > situation > > > > > > > > > > > >> as > > > > > > > > > > > >> > we > > > > > > > > > > > >> > >> >> are > > > > > > > > > > > >> > >> >> > now > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > > > different > > > > > > > > memory > > > > > > > > > > > pools > > > > > > > > > > > >> > are > > > > > > > > > > > >> > >> not > > > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > > > different > > > > > > > pool, > > > > > > > > > then > > > > > > > > > > > it > > > > > > > > > > > >> is > > > > > > > > > > > >> > >> quite > > > > > > > > > > > >> > >> >> > hard > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > process > > > to > > > > > get > > > > > > > > > killed > > > > > > > > > > > for > > > > > > > > > > > >> > using > > > > > > > > > > > >> > >> >> too > > > > > > > > > > > >> > >> >> > > much > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead > > to > > > a > > > > > > > similar > > > > > > > > > > > >> situation > > > > > > > > > > > >> > >> what > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > >> > >> >> > > have > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > setting > > > a > > > > > sane > > > > > > > > > default > > > > > > > > > > > >> value > > > > > > > > > > > >> > >> for > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > option > > > to > > > > > > > increase > > > > > > > > > it > > > > > > > > > > if > > > > > > > > > > > >> he > > > > > > > > > > > >> > >> runs > > > > > > > > > > > >> > >> >> into > > > > > > > > > > > >> > >> >> > > an > > > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > > > how > > > > > > > > would > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > >> > >> >> > lead > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > > 3 > > > > > > > > where > > > > > > > > > we > > > > > > > > > > > set > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> direct > > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On > > Fri, > > > > Aug > > > > > 9, > > > > > > > > 2019 > > > > > > > > > at > > > > > > > > > > > >> 9:12 > > > > > > > > > > > >> > AM > > > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > for > > > > > > the > > > > > > > > > > > feedback, > > > > > > > > > > > >> > >> Yang. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > > > your > > > > > > > > > > > comments: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > *Native > > > > > and > > > > > > > > Direct > > > > > > > > > > > >> Memory* > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > > think > > > > > > > setting > > > > > > > > a > > > > > > > > > > very > > > > > > > > > > > >> > large > > > > > > > > > > > >> > >> max > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > good > > > > > sides. > > > > > > > > E.g., > > > > > > > > > we > > > > > > > > > > > do > > > > > > > > > > > >> not > > > > > > > > > > > >> > >> >> worry > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > > > > allocate > > > > > > > > > managed > > > > > > > > > > / > > > > > > > > > > > >> > network > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() > . > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > However, > > > > > > there > > > > > > > > are > > > > > > > > > > > also > > > > > > > > > > > >> > some > > > > > > > > > > > >> > >> >> down > > > > > > > > > > > >> > >> >> > > sides > > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > One > > > > > > > thing I > > > > > > > > > can > > > > > > > > > > > >> think > > > > > > > > > > > >> > >> of is > > > > > > > > > > > >> > >> >> > that > > > > > > > > > > > >> > >> >> > > > if > > > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > killed > > > > > > due > > > > > > > to > > > > > > > > > > > >> overusing > > > > > > > > > > > >> > >> >> memory, > > > > > > > > > > > >> > >> >> > it > > > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > of > > > > the > > > > > > > memory > > > > > > > > > is > > > > > > > > > > > >> > overused. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > > > Another > > > > > > > > down > > > > > > > > > > side > > > > > > > > > > > >> is > > > > > > > > > > > >> > >> that > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > > memory > > > > > > > > > > > limit, > > > > > > > > > > > >> > >> because > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > means > > > > > we > > > > > > > kind > > > > > > > > > of > > > > > > > > > > > >> relay > > > > > > > > > > > >> > on > > > > > > > > > > > >> > >> >> heap > > > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > > > That > > > > > > > > > > could > > > > > > > > > > > >> be a > > > > > > > > > > > >> > >> >> problem > > > > > > > > > > > >> > >> >> > in > > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > usage > > > > > but > > > > > > > not > > > > > > > > > > > enough > > > > > > > > > > > >> > heap > > > > > > > > > > > >> > >> >> > activity > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Maybe > > > > you > > > > > > can > > > > > > > > > share > > > > > > > > > > > your > > > > > > > > > > > >> > >> reasons > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if > > > there > > > > > are > > > > > > > > > > anything > > > > > > > > > > > >> else > > > > > > > > > > > >> > I > > > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > > > Calculation* > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If > > > there > > > > > is > > > > > > > any > > > > > > > > > > > conflict > > > > > > > > > > > >> > >> between > > > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > > > specified, > > > > > > > > > > > I > > > > > > > > > > > >> > >> think we > > > > > > > > > > > >> > >> >> > > should > > > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > > think > > > > > > doing > > > > > > > > > > checking > > > > > > > > > > > >> on > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > client > > > > > > > > > > > >> > >> >> > > > side > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > K8s > > we > > > > can > > > > > > > > > discover > > > > > > > > > > > the > > > > > > > > > > > >> > >> problem > > > > > > > > > > > >> > >> >> > > before > > > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > > > > always > > > > > a > > > > > > > good > > > > > > > > > > > thing. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > But > > we > > > > can > > > > > > not > > > > > > > > > only > > > > > > > > > > > >> rely on > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > client > > > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > > > cluster > > > > > > > > > > > >> > >> TaskManagers > > > > > > > > > > > >> > >> >> on > > > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > > >> > client > > > > > > > > > > > >> > >> >> does > > > > > > > > > > > >> > >> >> > > see > > > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > What > > > do > > > > > you > > > > > > > > think? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On > > > Thu, > > > > > Aug > > > > > > 8, > > > > > > > > > 2019 > > > > > > > > > > at > > > > > > > > > > > >> 5:09 > > > > > > > > > > > >> > >> PM > > > > > > > > > > > >> > >> >> Yang > > > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > xintong, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > for > > > > > > > > your > > > > > > > > > > > >> detailed > > > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > > > it > > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > >> > more > > > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > just > > > > > have > > > > > > > few > > > > > > > > > > > >> questions > > > > > > > > > > > >> > >> about > > > > > > > > > > > >> > >> >> it. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Native > > > > > > > > and > > > > > > > > > > > Direct > > > > > > > > > > > >> > >> Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > > > do > > > > > not > > > > > > > > > > > >> differentiate > > > > > > > > > > > >> > >> user > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > included > > > > > > in > > > > > > > > task > > > > > > > > > > > >> off-heap > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Memory > > > > > > > > > > > >> Calculation > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > If > > > the > > > > > sum > > > > > > > of > > > > > > > > > and > > > > > > > > > > > >> > >> fine-grained > > > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > is > > > > > larger > > > > > > > than > > > > > > > > > > total > > > > > > > > > > > >> > >> process > > > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > we > > > > need > > > > > to > > > > > > > > check > > > > > > > > > > the > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > Song < > > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > everyone, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > > would > > > > > > > > like > > > > > > > > > to > > > > > > > > > > > >> start > > > > > > > > > > > >> > a > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > > > design > > > > > > > > > > > "Memory > > > > > > > > > > > >> > >> >> Management > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > updates > > > > > > > > > > from > > > > > > > > > > > >> > >> follow-up > > > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > This > > > > > > FLIP > > > > > > > > > > > addresses > > > > > > > > > > > >> > >> several > > > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Different > > > > > > > > > > > >> > >> configuration > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Complex > > > > > > > > > and > > > > > > > > > > > >> > >> difficult > > > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > Complicated, > > > > > > > > > > > >> > >> uncertain > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > > > changes > > > > > > > > to > > > > > > > > > > > solve > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > problems > > > > > > > > > > > >> > >> >> > > > can > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Extend > > > > > > > > > > memory > > > > > > > > > > > >> > >> manager > > > > > > > > > > > >> > >> >> to > > > > > > > > > > > >> > >> >> > > also > > > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Modify > > > > > > > > > how > > > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> reservations > > > > > > > > > > > >> > >> and > > > > > > > > > > > >> > >> >> > pools. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Simplify > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > > > find > > > > > > > > > more > > > > > > > > > > > >> > details > > > > > > > > > > > >> > >> in > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > > > note > > > > > > > > > > that > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> early > > > > > > > > > > > >> > >> >> > > design > > > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > > > to > > > > > > > > > > > have > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > > > forward > > > > > > > > > > to > > > > > > > > > > > >> your > > > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > > you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, > 2019 > > > at > > > > > 2:16 > > > > > > > PM > > > > > > > > > > > Xintong > > > > > > > > > > > >> > Song > > > > > > > > > > > >> > >> < > > > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for > sharing > > > > your > > > > > > > > opinion > > > > > > > > > > > Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in > favor > > of > > > > > > > > alternative > > > > > > > > > > 2. > > > > > > > > > > > I > > > > > > > > > > > >> was > > > > > > > > > > > >> > >> >> > wondering > > > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > > > Unsafe.allocate() > > > > > > for > > > > > > > > > > off-heap > > > > > > > > > > > >> > >> managed > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > > and > > > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. > But > > > > after > > > > > > > > giving > > > > > > > > > > it a > > > > > > > > > > > >> > second > > > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > > > >> > >> >> > > > I > > > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 > > using > > > > > direct > > > > > > > > > memory > > > > > > > > > > > for > > > > > > > > > > > >> > >> off-heap > > > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > > > concern, > > > > > I > > > > > > > > think > > > > > > > > > > what > > > > > > > > > > > >> > >> proposed > > > > > > > > > > > >> > >> >> in > > > > > > > > > > > >> > >> >> > > this > > > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap managed > > > > memory > > > > > > and > > > > > > > > > > network > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Unsafe.allocate(), > > > > which > > > > > > > means > > > > > > > > > > they > > > > > > > > > > > >> are > > > > > > > > > > > >> > >> >> > practically > > > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM > max > > > > > direct > > > > > > > > > memory. > > > > > > > > > > > The > > > > > > > > > > > >> > only > > > > > > > > > > > >> > >> >> parts > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct > memory > > > are > > > > > task > > > > > > > > > > off-heap > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> and > > > > > > > > > > > >> > >> >> > JVM > > > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly > > alternative > > > 2 > > > > > > > suggests > > > > > > > > > to > > > > > > > > > > > set > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> JVM > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > 1:48 > > > > > > > > PM > > > > > > > > > > Till > > > > > > > > > > > >> > >> Rohrmann > > > > > > > > > > > >> > >> >> < > > > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for the > > > > > > > clarification > > > > > > > > > > > >> Xintong. I > > > > > > > > > > > >> > >> >> > > understand > > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be in > > > favour > > > > > of > > > > > > > > > option 2 > > > > > > > > > > > >> > because > > > > > > > > > > > >> > >> it > > > > > > > > > > > >> > >> >> > makes > > > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit > the > > > > direct > > > > > > > > > memory, I > > > > > > > > > > > >> fear > > > > > > > > > > > >> > >> that > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > >> > >> >> > > might > > > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as > we > > > are > > > > > > > > currently > > > > > > > > > > in: > > > > > > > > > > > >> The > > > > > > > > > > > >> > >> user > > > > > > > > > > > >> > >> >> > might > > > > > > > > > > > >> > >> >> > > > see > > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by the > OS > > > and > > > > > > does > > > > > > > > not > > > > > > > > > > know > > > > > > > > > > > >> why > > > > > > > > > > > >> > >> this > > > > > > > > > > > >> > >> >> is > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > decrease > > > > the > > > > > > > > process > > > > > > > > > > > memory > > > > > > > > > > > >> > size > > > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in > order > > to > > > > > > > > accommodate > > > > > > > > > > for > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> extra > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > decrease > > > > > memory > > > > > > > > > budgets > > > > > > > > > > > >> which > > > > > > > > > > > >> > >> are > > > > > > > > > > > >> > >> >> not > > > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the > > overall > > > > > > memory > > > > > > > > > > > >> consumption. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug > 16, > > > 2019 > > > > > at > > > > > > > > 11:01 > > > > > > > > > AM > > > > > > > > > > > >> > Xintong > > > > > > > > > > > >> > >> >> Song < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me > explain > > > > this > > > > > > > with a > > > > > > > > > > > >> concrete > > > > > > > > > > > >> > >> >> example > > > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say we > > > have > > > > > the > > > > > > > > > > following > > > > > > > > > > > >> > >> scenario. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total > Process > > > > > Memory: > > > > > > > 1GB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > > > Memory > > > > > > (Task > > > > > > > > > > Off-Heap > > > > > > > > > > > >> > >> Memory + > > > > > > > > > > > >> > >> >> JVM > > > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other Memory > > > (JVM > > > > > Heap > > > > > > > > > Memory, > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > Memory): > > > > > 800MB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > alternative > > > 2, > > > > > we > > > > > > > set > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > alternative > > > 3, > > > > > we > > > > > > > set > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say > 1TB. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > actual > > > > direct > > > > > > > > memory > > > > > > > > > > > usage > > > > > > > > > > > >> of > > > > > > > > > > > >> > >> Task > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not > exceed > > > > 200MB, > > > > > > > then > > > > > > > > > > > >> > alternative 2 > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same > utility. > > > > > Setting > > > > > > > > larger > > > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of the > > > other > > > > > > > memory > > > > > > > > > > pools. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > actual > > > > direct > > > > > > > > memory > > > > > > > > > > > usage > > > > > > > > > > > >> of > > > > > > > > > > > >> > >> Task > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > > > potentially > > > > > > > > exceed > > > > > > > > > > > 200MB, > > > > > > > > > > > >> > then > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > > > Alternative 2 > > > > > > > suffers > > > > > > > > > > from > > > > > > > > > > > >> > >> frequent > > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > > >> > >> >> > > To > > > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user can > do > > > is > > > > to > > > > > > > > modify > > > > > > > > > > the > > > > > > > > > > > >> > >> >> configuration > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > > > Off-Heap > > > > > > > Memory + > > > > > > > > > JVM > > > > > > > > > > > >> > >> Overhead). > > > > > > > > > > > >> > >> >> > Let's > > > > > > > > > > > >> > >> >> > > > say > > > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct > > Memory > > > > to > > > > > > > 250MB, > > > > > > > > > > this > > > > > > > > > > > >> will > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools to > > > 750MB, > > > > > > given > > > > > > > > the > > > > > > > > > > > total > > > > > > > > > > > >> > >> process > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > > > alternative > > > > > > 3, > > > > > > > > > there > > > > > > > > > > is > > > > > > > > > > > >> no > > > > > > > > > > > >> > >> >> chance of > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of > > exceeding > > > > the > > > > > > > total > > > > > > > > > > > process > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > limit, > > > > > > > > > > > >> > >> >> > > > but > > > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use > up > > > all > > > > > the > > > > > > > > > reserved > > > > > > > > > > > >> native > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, > JVM > > > > > > > Metaspace), > > > > > > > > > if > > > > > > > > > > > the > > > > > > > > > > > >> > >> actual > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet very > > > close > > > > to > > > > > > > > 200MB, > > > > > > > > > > user > > > > > > > > > > > >> > >> probably > > > > > > > > > > > >> > >> >> do > > > > > > > > > > > >> > >> >> > > not > > > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Therefore, I > > > think > > > > > > from > > > > > > > > the > > > > > > > > > > > user's > > > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for > > alternative > > > 2 > > > > > may > > > > > > > lead > > > > > > > > > to > > > > > > > > > > > >> lower > > > > > > > > > > > >> > >> >> resource > > > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative > 3. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, Aug > > 16, > > > > 2019 > > > > > > at > > > > > > > > > 10:28 > > > > > > > > > > AM > > > > > > > > > > > >> Till > > > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > > > >> > >> >> > > < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess > you > > > have > > > > > to > > > > > > > help > > > > > > > > > me > > > > > > > > > > > >> > >> understand > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > alternative > 2 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 wrt > to > > > > > memory > > > > > > > > under > > > > > > > > > > > >> > utilization > > > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > Alternative > > > 2: > > > > > set > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Overhead. > > Then > > > > > there > > > > > > > is > > > > > > > > > the > > > > > > > > > > > risk > > > > > > > > > > > >> > that > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > >> > >> >> > > size > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of > > garbage > > > > > > > > collection > > > > > > > > > > and > > > > > > > > > > > >> > >> >> potentially > > > > > > > > > > > >> > >> >> > an > > > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > Alternative > > > 3: > > > > > set > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > alternative > > 2. > > > > > This > > > > > > > > would > > > > > > > > > of > > > > > > > > > > > >> course > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How would > > > > > > alternative > > > > > > > 2 > > > > > > > > > now > > > > > > > > > > > >> result > > > > > > > > > > > >> > >> in an > > > > > > > > > > > >> > >> >> > > under > > > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared > to > > > > > > > alternative > > > > > > > > 3? > > > > > > > > > > If > > > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > memory > > > > size > > > > > > and > > > > > > > > we > > > > > > > > > > use > > > > > > > > > > > >> only > > > > > > > > > > > >> > >> >> little, > > > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > alternative > > 3 > > > > > > results > > > > > > > in > > > > > > > > > > > memory > > > > > > > > > > > >> > under > > > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, > Aug > > > 13, > > > > > 2019 > > > > > > > at > > > > > > > > > 4:19 > > > > > > > > > > > PM > > > > > > > > > > > >> > Yang > > > > > > > > > > > >> > >> >> Wang < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > > > xintong,till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Native > > and > > > > > > Direct > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My point > > is > > > > > > setting > > > > > > > a > > > > > > > > > very > > > > > > > > > > > >> large > > > > > > > > > > > >> > >> max > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > differentiate > > > > > > direct > > > > > > > > and > > > > > > > > > > > >> native > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > >> > >> >> > If > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory,including > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > > > memory > > > > > and > > > > > > > > > > framework > > > > > > > > > > > >> > direct > > > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am in > > > favor > > > > of > > > > > > > > setting > > > > > > > > > > > >> direct > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > with > > > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Memory > > > > > > Calculation > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I agree > > with > > > > > > > xintong. > > > > > > > > > For > > > > > > > > > > > Yarn > > > > > > > > > > > >> > and > > > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > > > >> > >> >> > > > need > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > configurations > > > > > in > > > > > > > > client > > > > > > > > > > to > > > > > > > > > > > >> avoid > > > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the > flink > > > > > master. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Xintong > > > Song < > > > > > > > > > > > >> > >> [hidden email] > > > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thanks > > for > > > > > > > replying, > > > > > > > > > > Till. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > > > > > > MemorySegment, I > > > > > > > > > > > think > > > > > > > > > > > >> > you > > > > > > > > > > > >> > >> are > > > > > > > > > > > >> > >> >> > > right > > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > issue > in > > > the > > > > > > scope > > > > > > > > of > > > > > > > > > > this > > > > > > > > > > > >> > FLIP. > > > > > > > > > > > >> > >> >> This > > > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > configure > > > > > memory > > > > > > > > pools > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory > > > > > consumers > > > > > > > use > > > > > > > > > it. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > About > > > direct > > > > > > > > memory, I > > > > > > > > > > > think > > > > > > > > > > > >> > >> >> > alternative > > > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > reservation > > > > > > issue > > > > > > > > that > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > >> > >> >> > > does, > > > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > using > > > memory > > > > > at > > > > > > > the > > > > > > > > > > > >> container > > > > > > > > > > > >> > >> level, > > > > > > > > > > > >> > >> >> > > which > > > > > > > > > > > >> > >> >> > > > is > > > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both > > "Task > > > > > > > Off-Heap > > > > > > > > > > > Memory" > > > > > > > > > > > >> and > > > > > > > > > > > >> > >> "JVM > > > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > alternative > > > > 2, > > > > > > > users > > > > > > > > > > might > > > > > > > > > > > >> > >> configure > > > > > > > > > > > >> > >> >> > them > > > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just > to > > > > avoid > > > > > > > > getting > > > > > > > > > a > > > > > > > > > > > >> direct > > > > > > > > > > > >> > >> OOM. > > > > > > > > > > > >> > >> >> For > > > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > OOM, > > > > so > > > > > > > they > > > > > > > > > may > > > > > > > > > > > not > > > > > > > > > > > >> > >> config > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > two > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > > > consequences > > > > > > > are > > > > > > > > > > risks > > > > > > > > > > > >> of > > > > > > > > > > > >> > >> >> overall > > > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Thank > > you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Xintong > > > Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On > Tue, > > > Aug > > > > > 13, > > > > > > > 2019 > > > > > > > > > at > > > > > > > > > > > >> 9:39 AM > > > > > > > > > > > >> > >> Till > > > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > > > > proposing > > > > > > > > > > > this > > > > > > > > > > > >> > FLIP > > > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > All > in > > > > all I > > > > > > > think > > > > > > > > > it > > > > > > > > > > > >> already > > > > > > > > > > > >> > >> >> looks > > > > > > > > > > > >> > >> >> > > quite > > > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > question > > > > > about > > > > > > > > > > > allocating > > > > > > > > > > > >> > >> memory > > > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > strictly > > > > > > > necessary > > > > > > > > > to > > > > > > > > > > do > > > > > > > > > > > >> in > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > context > > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > > done > > > > as a > > > > > > > > follow > > > > > > > > > > up? > > > > > > > > > > > >> > Without > > > > > > > > > > > >> > >> >> > knowing > > > > > > > > > > > >> > >> >> > > > all > > > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > that > > we > > > > > would > > > > > > > > widen > > > > > > > > > > the > > > > > > > > > > > >> scope > > > > > > > > > > > >> > >> of > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to > > touch > > > > all > > > > > > the > > > > > > > > > > > existing > > > > > > > > > > > >> > call > > > > > > > > > > > >> > >> >> sites > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > memory > > > > > > segments > > > > > > > > > (this > > > > > > > > > > > >> should > > > > > > > > > > > >> > >> >> mainly > > > > > > > > > > > >> > >> >> > be > > > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > memory > > > > > > > > > reservation > > > > > > > > > > > >> call > > > > > > > > > > > >> > to > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > this > > > and I > > > > > > would > > > > > > > > > hope > > > > > > > > > > > that > > > > > > > > > > > >> > >> this is > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > streaming > > > > > job > > > > > > > > would > > > > > > > > > > have > > > > > > > > > > > >> with > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Concerning > > > > > the > > > > > > > > > second > > > > > > > > > > > open > > > > > > > > > > > >> > >> >> question > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > direct > > > > > memory > > > > > > > > > limit, I > > > > > > > > > > > >> would > > > > > > > > > > > >> > >> also > > > > > > > > > > > >> > >> >> be > > > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > leaving > > > it > > > > > > open > > > > > > > > > would > > > > > > > > > > be > > > > > > > > > > > >> > best. > > > > > > > > > > > >> > >> My > > > > > > > > > > > >> > >> >> > > concern > > > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > in > > a > > > > > > similar > > > > > > > > > > > situation > > > > > > > > > > > >> as > > > > > > > > > > > >> > we > > > > > > > > > > > >> > >> >> are > > > > > > > > > > > >> > >> >> > now > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > the > > > > > different > > > > > > > > memory > > > > > > > > > > > pools > > > > > > > > > > > >> > are > > > > > > > > > > > >> > >> not > > > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > > > different > > > > > > > pool, > > > > > > > > > then > > > > > > > > > > > it > > > > > > > > > > > >> is > > > > > > > > > > > >> > >> quite > > > > > > > > > > > >> > >> >> > hard > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > process > > > to > > > > > get > > > > > > > > > killed > > > > > > > > > > > for > > > > > > > > > > > >> > using > > > > > > > > > > > >> > >> >> too > > > > > > > > > > > >> > >> >> > > much > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > lead > > to > > > a > > > > > > > similar > > > > > > > > > > > >> situation > > > > > > > > > > > >> > >> what > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > >> > >> >> > > have > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > setting > > > a > > > > > sane > > > > > > > > > default > > > > > > > > > > > >> value > > > > > > > > > > > >> > >> for > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > option > > > to > > > > > > > increase > > > > > > > > > it > > > > > > > > > > if > > > > > > > > > > > >> he > > > > > > > > > > > >> > >> runs > > > > > > > > > > > >> > >> >> into > > > > > > > > > > > >> > >> >> > > an > > > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > > > how > > > > > > > > would > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > >> > >> >> > lead > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > > 3 > > > > > > > > where > > > > > > > > > we > > > > > > > > > > > set > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> direct > > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On > > Fri, > > > > Aug > > > > > 9, > > > > > > > > 2019 > > > > > > > > > at > > > > > > > > > > > >> 9:12 > > > > > > > > > > > >> > AM > > > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > for > > > > > > the > > > > > > > > > > > feedback, > > > > > > > > > > > >> > >> Yang. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > > > your > > > > > > > > > > > comments: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > *Native > > > > > and > > > > > > > > Direct > > > > > > > > > > > >> Memory* > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > > think > > > > > > > setting > > > > > > > > a > > > > > > > > > > very > > > > > > > > > > > >> > large > > > > > > > > > > > >> > >> max > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > good > > > > > sides. > > > > > > > > E.g., > > > > > > > > > we > > > > > > > > > > > do > > > > > > > > > > > >> not > > > > > > > > > > > >> > >> >> worry > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > to > > > > > allocate > > > > > > > > > managed > > > > > > > > > > / > > > > > > > > > > > >> > network > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > >> > >> >> > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Unsafe.allocate() > . > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > However, > > > > > > there > > > > > > > > are > > > > > > > > > > > also > > > > > > > > > > > >> > some > > > > > > > > > > > >> > >> >> down > > > > > > > > > > > >> > >> >> > > sides > > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > One > > > > > > > thing I > > > > > > > > > can > > > > > > > > > > > >> think > > > > > > > > > > > >> > >> of is > > > > > > > > > > > >> > >> >> > that > > > > > > > > > > > >> > >> >> > > > if > > > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > killed > > > > > > due > > > > > > > to > > > > > > > > > > > >> overusing > > > > > > > > > > > >> > >> >> memory, > > > > > > > > > > > >> > >> >> > it > > > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > of > > > > the > > > > > > > memory > > > > > > > > > is > > > > > > > > > > > >> > overused. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > - > > > > > Another > > > > > > > > down > > > > > > > > > > side > > > > > > > > > > > >> is > > > > > > > > > > > >> > >> that > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > > memory > > > > > > > > > > > limit, > > > > > > > > > > > >> > >> because > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > means > > > > > we > > > > > > > kind > > > > > > > > > of > > > > > > > > > > > >> relay > > > > > > > > > > > >> > on > > > > > > > > > > > >> > >> >> heap > > > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > > > That > > > > > > > > > > could > > > > > > > > > > > >> be a > > > > > > > > > > > >> > >> >> problem > > > > > > > > > > > >> > >> >> > in > > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > usage > > > > > but > > > > > > > not > > > > > > > > > > > enough > > > > > > > > > > > >> > heap > > > > > > > > > > > >> > >> >> > activity > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Maybe > > > > you > > > > > > can > > > > > > > > > share > > > > > > > > > > > your > > > > > > > > > > > >> > >> reasons > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > if > > > there > > > > > are > > > > > > > > > > anything > > > > > > > > > > > >> else > > > > > > > > > > > >> > I > > > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > > > Calculation* > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > If > > > there > > > > > is > > > > > > > any > > > > > > > > > > > conflict > > > > > > > > > > > >> > >> between > > > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > > > specified, > > > > > > > > > > > I > > > > > > > > > > > >> > >> think we > > > > > > > > > > > >> > >> >> > > should > > > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > I > > > think > > > > > > doing > > > > > > > > > > checking > > > > > > > > > > > >> on > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > client > > > > > > > > > > > >> > >> >> > > > side > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > K8s > > we > > > > can > > > > > > > > > discover > > > > > > > > > > > the > > > > > > > > > > > >> > >> problem > > > > > > > > > > > >> > >> >> > > before > > > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > is > > > > always > > > > > a > > > > > > > good > > > > > > > > > > > thing. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > But > > we > > > > can > > > > > > not > > > > > > > > > only > > > > > > > > > > > >> rely on > > > > > > > > > > > >> > >> the > > > > > > > > > > > >> > >> >> > > client > > > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > > > cluster > > > > > > > > > > > >> > >> TaskManagers > > > > > > > > > > > >> > >> >> on > > > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > > >> > client > > > > > > > > > > > >> > >> >> does > > > > > > > > > > > >> > >> >> > > see > > > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > What > > > do > > > > > you > > > > > > > > think? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > On > > > Thu, > > > > > Aug > > > > > > 8, > > > > > > > > > 2019 > > > > > > > > > > at > > > > > > > > > > > >> 5:09 > > > > > > > > > > > >> > >> PM > > > > > > > > > > > >> > >> >> Yang > > > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > xintong, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > for > > > > > > > > your > > > > > > > > > > > >> detailed > > > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > > > it > > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > >> > more > > > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > just > > > > > have > > > > > > > few > > > > > > > > > > > >> questions > > > > > > > > > > > >> > >> about > > > > > > > > > > > >> > >> >> it. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Native > > > > > > > > and > > > > > > > > > > > Direct > > > > > > > > > > > >> > >> Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > We > > > do > > > > > not > > > > > > > > > > > >> differentiate > > > > > > > > > > > >> > >> user > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > included > > > > > > in > > > > > > > > task > > > > > > > > > > > >> off-heap > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Memory > > > > > > > > > > > >> Calculation > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > If > > > the > > > > > sum > > > > > > > of > > > > > > > > > and > > > > > > > > > > > >> > >> fine-grained > > > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > is > > > > > larger > > > > > > > than > > > > > > > > > > total > > > > > > > > > > > >> > >> process > > > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > we > > > > need > > > > > to > > > > > > > > check > > > > > > > > > > the > > > > > > > > > > > >> > memory > > > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > Song < > > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > everyone, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > > would > > > > > > > > like > > > > > > > > > to > > > > > > > > > > > >> start > > > > > > > > > > > >> > a > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > > > design > > > > > > > > > > > "Memory > > > > > > > > > > > >> > >> >> Management > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > updates > > > > > > > > > > from > > > > > > > > > > > >> > >> follow-up > > > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > This > > > > > > FLIP > > > > > > > > > > > addresses > > > > > > > > > > > >> > >> several > > > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Different > > > > > > > > > > > >> > >> configuration > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Complex > > > > > > > > > and > > > > > > > > > > > >> > >> difficult > > > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > Complicated, > > > > > > > > > > > >> > >> uncertain > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > > > changes > > > > > > > > to > > > > > > > > > > > solve > > > > > > > > > > > >> > the > > > > > > > > > > > >> > >> >> > problems > > > > > > > > > > > >> > >> >> > > > can > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Extend > > > > > > > > > > memory > > > > > > > > > > > >> > >> manager > > > > > > > > > > > >> > >> >> to > > > > > > > > > > > >> > >> >> > > also > > > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Modify > > > > > > > > > how > > > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > >> reservations > > > > > > > > > > > >> > >> and > > > > > > > > > > > >> > >> >> > pools. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Simplify > > > > > > > > > > > memory > > > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > > > find > > > > > > > > > more > > > > > > > > > > > >> > details > > > > > > > > > > > >> > >> in > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > > > note > > > > > > > > > > that > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> early > > > > > > > > > > > >> > >> >> > > design > > > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > > > to > > > > > > > > > > > have > > > > > > > > > > > >> the > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > > > forward > > > > > > > > > > to > > > > > > > > > > > >> your > > > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > > you~ > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > Song > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS] FLIP-49: Unified Memory Configuration for TaskExecutors
|
|
@Xintong @Till
Thanks for more thoughts. Just for clarification, to make querying of segments oblivious to the memory type for Dataset jobs, we currently need that some component keeps the slot/owner limit for each memory type and allocates segments up to that limit for each task and type. Otherwise, the whole Dataset code has to be refactored to query the memory pools separately up to their limit. If we keep memory manager per TM then this is one more concern that has to be added to the memory manager right now: memory tracking per owner which we might do later for revocation but to me it still looks like a bigger change and design decision comparing to just changing scope between TM and slot. Best, Andrey On Mon, Sep 16, 2019 at 2:21 PM Xintong Song <[hidden email]> wrote: > Fair enough. > > Thank you~ > > Xintong Song > > > > On Mon, Sep 16, 2019 at 7:28 PM Till Rohrmann <[hidden email]> > wrote: > > > Yes this could be the case. But from the point of the slot, it doesn't > > really matter to it because it is abstracted away and can be added at a > > later time when really needed. > > > > Cheers, > > Till > > > > On Mon, Sep 16, 2019 at 11:42 AM Xintong Song <[hidden email]> > > wrote: > > > > > @Till > > > > > > Just trying to understand. Are you suggesting that we have both a task > > > executor level memory manager, and multiple slot level memory managers > > with > > > in the task executor level one? (Or what ever the components are > called) > > > > > > If that is what you mean, then it makes a lot sense to me. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:30 PM Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > I think conceptually it makes sense to have a MemoryManager per slot. > > As > > > > Andrey said, it should not add any noticeable overhead. Moreover, I > > think > > > > this model does not contradict any of the mentioned future > > improvements. > > > > For example, a per slot MemoryManager could still be implemented such > > > that > > > > the TaskExecutor's MemoryManager keeps a reference to it and can > > request > > > to > > > > give up/assign more memory segments in case of revocable memory. The > > > > benefit I see is a better abstraction and hence a more targeted > > component > > > > which will be easier to maintain and evolve. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Mon, Sep 16, 2019 at 11:23 AM Xintong Song <[hidden email] > > > > > > wrote: > > > > > > > > > I would say both ways (one unified memory manager per task > executor, > > or > > > > one > > > > > isolated memory manager per slot) should work well at the moment. > > I've > > > > not > > > > > yet think this through very detailedly, but just from what I can > > think > > > of > > > > > at the moment, I'm in favor of having one memory manager per task > > > > executor, > > > > > for the following reasons: > > > > > > > > > > - We need the memory manager to understand quota of each operator > > > anyway > > > > > (with the provided fractions), for limiting the memory usage of > > > > > un-revocable operators. It shouldn't be much more complicated for > the > > > > > memory manager to isolate memory across slots. Actually I'm > wondering > > > > > wether it requires more efforts at all compare to having one memory > > > > manager > > > > > per slot, where we also need to handle the creation and destroying > of > > > > > memory managers. > > > > > > > > > > - It is true that we have no plan for segments caching at the > moment. > > > > But I > > > > > would say it is a good opportunity for future improvement. E.g., we > > may > > > > > have some "lazy de-allocation" strategy that only de-allocate > > segments > > > if > > > > > the remaining available memory quota cannot satisfy new coming > > > > reservation > > > > > requests. I understand that we should not complicate the current > > design > > > > for > > > > > any "potential future features". My concern is that we make efforts > > > > > changing from one memory manager per task executor to one per slot, > > > with > > > > > the benefit (easier to isolate memory across slots) that is > uncertain > > > to > > > > > me, and later discover that we may want to change it back for > further > > > > > optimizations. > > > > > > > > > > Anyway, I do not have strong objection against having one memory > > > manager > > > > > per slot. Just raising my concerns. > > > > > > > > > > Thank you~ > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > On Mon, Sep 16, 2019 at 4:16 PM Andrey Zagrebin < > > [hidden email]> > > > > > wrote: > > > > > > > > > > > @Xintong > > > > > > > > > > > > Just to understand, what kind of overhead do you mean by creating > > the > > > > > > memory manager for every dynamic slot? > > > > > > We agreed not to preallocate segments anymore and there was no > plan > > > to > > > > do > > > > > > any segment caching at the moment, AFAIK. > > > > > > This means the segments are just garbage collected upon release > and > > > not > > > > > > reused. > > > > > > If we decide to go with caching or revoking later and share pools > > > among > > > > > > slots, it would already mean mixed segment/reservation types of > > > memory > > > > > > allocation in general (like batch and streaming). > > > > > > Then I think memory manager would need some input from tasks > > because > > > it > > > > > is > > > > > > hard for Flink to decide how much to cache and which pool to > > allocate > > > > > from > > > > > > eventually. > > > > > > As I understand, tasks do not really need this now. That is why I > > > think > > > > > > having memory manager per slot will give more security by > isolating > > > > slots > > > > > > with relatively small effort. > > > > > > If there is no preallocation/caching then creating the memory > > manager > > > > is > > > > > > almost no-op, later we can reconsider it if needed. What do you > > > think? > > > > > > > > > > > > Thanks, > > > > > > Andrey > > > > > > > > > > > > On Sat, Sep 14, 2019 at 8:11 AM Xintong Song < > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > @Andrey, > > > > > > > > > > > > > > If we speak in the condition of dynamic slot allocation, then > > > DataSet > > > > > > tasks > > > > > > > should always request slots with unknown resource profiles. > Then > > > the > > > > > > > allocated slots should have 1/n on-heap managed memory and 1/n > > > > off-heap > > > > > > > managed memory of the task executor, where n is the configured > > > > > > > 'numberOfSlot'. Given that DataSet operators only allocate > > segments > > > > (no > > > > > > > reservation), and does not care whether the segments are > on-heap > > or > > > > > > > off-heap, I think it makes sense to wrapping both pools for > > DataSet > > > > > > > operators, with respect to the on-heap/off-heap quota of the > > > > allocated > > > > > > > slot. > > > > > > > > > > > > > > I'm not sure whether we want to create one memory manager per > > > slot. I > > > > > > think > > > > > > > we can have limit the quota of each slot with one memory > manager > > > per > > > > > task > > > > > > > executor. Given the dynamic slot allocation that dynamically > > create > > > > and > > > > > > > destroy slots, I think having one memory manager per task > > executor > > > > may > > > > > > save > > > > > > > the overhead for frequently creating and destroying memory > > > managers, > > > > > and > > > > > > > provide the chance for reusing memory segments across slots. > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 11:47 PM Andrey Zagrebin < > > > > [hidden email] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > > > True, there would be no regression if only one type of memory > > is > > > > > > > > configured. This can be a problem only for the old jobs > running > > > in > > > > a > > > > > > > newly > > > > > > > > configured cluster. > > > > > > > > > > > > > > > > About the pool type precedence, in general, it should not > > matter > > > > for > > > > > > the > > > > > > > > users which type the segments have. > > > > > > > > The first implementation can be just to pull from any pool, > > e.g. > > > > > empty > > > > > > > one > > > > > > > > pool firstly and then another or some other random pulling. > > > > > > > > This might be a problem if we mix segment allocations and > > > > reservation > > > > > > of > > > > > > > > memory chunks from the same memory manager. > > > > > > > > The reservation will be usually for a certain type of memory > > then > > > > the > > > > > > > task > > > > > > > > will probably have to also decide from which pool to allocate > > the > > > > > > > segments. > > > > > > > > I would suggest we create a memory manager per slot and give > it > > > > > memory > > > > > > > > limit of the slot then we do not have this kind of mixed > > > operation > > > > > > > > because Dataset/Batch jobs need only segment memory > allocations > > > and > > > > > > > > streaming jobs need only memory chunks for state backends as > I > > > > > > understand > > > > > > > > the current plan. > > > > > > > > I would suggest we will look at it if we have the mixed > > > operations > > > > at > > > > > > > some > > > > > > > > point and it becomes a problem. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrey > > > > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 5:24 PM Andrey Zagrebin < > > > > > [hidden email]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > > > > From: Xintong Song <[hidden email]> > > > > > > > > > Date: Thu, Sep 12, 2019 at 4:21 AM > > > > > > > > > Subject: Re: [DISCUSS] FLIP-49: Unified Memory > Configuration > > > for > > > > > > > > > TaskExecutors > > > > > > > > > To: dev <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Andrey, > > > > > > > > > > > > > > > > > > Thanks for bringing this up. > > > > > > > > > > > > > > > > > > If I understand correctly, this issue only occurs where the > > > > cluster > > > > > > is > > > > > > > > > configured with both on-heap and off-heap memory. There > > should > > > be > > > > > no > > > > > > > > > regression for clusters configured in the old way (either > all > > > > > on-heap > > > > > > > or > > > > > > > > > all off-heap). > > > > > > > > > > > > > > > > > > I also agree that it would be good if the DataSet API jobs > > can > > > > use > > > > > > both > > > > > > > > > memory types. The only question I can see is that, from > which > > > > pool > > > > > > > (heap > > > > > > > > / > > > > > > > > > off-heap) should we allocate memory for DataSet API > > operators? > > > Do > > > > > we > > > > > > > > > always prioritize one pool over the other? Or do we always > > > > > prioritize > > > > > > > the > > > > > > > > > pool with more available memory left? > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 8:15 PM Andrey Zagrebin < > > > > > > [hidden email]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > While looking more into the implementation details of > Step > > 4, > > > > we > > > > > > > > released > > > > > > > > > > during some offline discussions with @Till > > > > > > > > > > that there can be a performance degradation for the batch > > > > DataSet > > > > > > API > > > > > > > > if > > > > > > > > > we > > > > > > > > > > simply continue to pull memory from the pool > > > > > > > > > > according the legacy option taskmanager.memory.off-heap. > > > > > > > > > > > > > > > > > > > > The reason is that if the cluster is newly configured to > > > > > statically > > > > > > > > split > > > > > > > > > > heap/off-heap (not like previously either heap or > 0ff-heap) > > > > > > > > > > then the batch DataSet API jobs will be able to use only > > one > > > > type > > > > > > of > > > > > > > > > > memory. Although it does not really matter where the > memory > > > > > > segments > > > > > > > > come > > > > > > > > > > from > > > > > > > > > > and potentially batch jobs can use both. Also, currently > > the > > > > > > Dataset > > > > > > > > API > > > > > > > > > > does not result in absolute resource requirements and its > > > batch > > > > > > jobs > > > > > > > > will > > > > > > > > > > always get a default share of TM resources. > > > > > > > > > > > > > > > > > > > > The suggestion is that we let the batch tasks of Dataset > > API > > > > pull > > > > > > > from > > > > > > > > > both > > > > > > > > > > pools according to their fair slot share of each memory > > type. > > > > > > > > > > For that we can have a special wrapping view of both > pools > > > > which > > > > > > will > > > > > > > > > pull > > > > > > > > > > segments (can be randomly) according to the slot limits. > > > > > > > > > > The view can wrap TM level memory pools and be given to > the > > > > Task. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 1:35 PM Xintong Song < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for your comments, Andrey. > > > > > > > > > > > > > > > > > > > > > > - Regarding Task Off-Heap Memory, I think you're right > > that > > > > the > > > > > > > user > > > > > > > > > need > > > > > > > > > > > to make sure that direct memory and native memory > > together > > > > used > > > > > > by > > > > > > > > the > > > > > > > > > > user > > > > > > > > > > > code (external libs) do not exceed the configured > value. > > As > > > > far > > > > > > as > > > > > > > I > > > > > > > > > can > > > > > > > > > > > think of, there is nothing we can do about it. > > > > > > > > > > > > > > > > > > > > > > I addressed the rest of your comment in the wiki page > > [1]. > > > > > Please > > > > > > > > take > > > > > > > > > a > > > > > > > > > > > look. > > > > > > > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 6:13 PM Andrey Zagrebin < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > EDIT: sorry for confusion I meant > > > > > > > > > > > > taskmanager.memory.off-heap > > > > > > > > > > > > instead of > > > > > > > > > > > > setting taskmanager.memory.preallocate > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 11:29 AM Andrey Zagrebin < > > > > > > > > > [hidden email]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > @Xitong thanks a lot for driving the discussion. > > > > > > > > > > > > > > > > > > > > > > > > > > I also reviewed the FLIP and it looks quite good to > > me. > > > > > > > > > > > > > Here are some comments: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - One thing I wanted to discuss is the > > > > > > > backwards-compatibility > > > > > > > > > > with > > > > > > > > > > > > > the previous user setups. We could list which > > > options > > > > we > > > > > > > plan > > > > > > > > to > > > > > > > > > > > > deprecate. > > > > > > > > > > > > > From the first glance it looks possible to > provide > > > the > > > > > > > > > > same/similar > > > > > > > > > > > > > behaviour for the setups relying on the > deprecated > > > > > > options. > > > > > > > > E.g. > > > > > > > > > > > > > setting taskmanager.memory.preallocate to true > > could > > > > > > > override > > > > > > > > > the > > > > > > > > > > > > > new taskmanager.memory.managed.offheap-fraction > > to 1 > > > > > etc. > > > > > > At > > > > > > > > the > > > > > > > > > > > > moment the > > > > > > > > > > > > > FLIP just states that in some cases it may > require > > > > > > > > > re-configuring > > > > > > > > > > of > > > > > > > > > > > > > cluster if migrated from prior versions. My > > > suggestion > > > > > is > > > > > > > that > > > > > > > > > we > > > > > > > > > > > try > > > > > > > > > > > > to > > > > > > > > > > > > > keep it backwards-compatible unless there is a > > good > > > > > reason > > > > > > > > like > > > > > > > > > > some > > > > > > > > > > > > major > > > > > > > > > > > > > complication for the implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also couple of smaller things: > > > > > > > > > > > > > > > > > > > > > > > > > > - I suggest we remove TaskExecutorSpecifics from > > the > > > > > FLIP > > > > > > > and > > > > > > > > > > leave > > > > > > > > > > > > > some general wording atm, like 'data structure > to > > > > store' > > > > > > or > > > > > > > > > > 'utility > > > > > > > > > > > > > classes'. When the classes are implemented, we > put > > > the > > > > > > > > concrete > > > > > > > > > > > class > > > > > > > > > > > > > names. This way we can avoid confusion and stale > > > > > > documents. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - As I understand, if user task uses native > memory > > > > (not > > > > > > > direct > > > > > > > > > > > memory, > > > > > > > > > > > > > but e.g. unsafe.allocate or from external lib), > > > there > > > > > will > > > > > > > be > > > > > > > > no > > > > > > > > > > > > > explicit guard against exceeding 'task off heap > > > > memory'. > > > > > > > Then > > > > > > > > > user > > > > > > > > > > > > should > > > > > > > > > > > > > still explicitly make sure that her/his direct > > > buffer > > > > > > > > allocation > > > > > > > > > > > plus > > > > > > > > > > > > any > > > > > > > > > > > > > other memory usages does not exceed value > > announced > > > as > > > > > > 'task > > > > > > > > off > > > > > > > > > > > > heap'. I > > > > > > > > > > > > > guess there is no so much that can be done about > > it > > > > > except > > > > > > > > > > > mentioning > > > > > > > > > > > > in > > > > > > > > > > > > > docs, similar to controlling the heap state > > backend. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Andrey > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 2, 2019 at 10:07 AM Yang Wang < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> I also agree that all the configuration should be > > > > > calculated > > > > > > > out > > > > > > > > > of > > > > > > > > > > > > >> TaskManager. > > > > > > > > > > > > >> > > > > > > > > > > > > >> So a full configuration should be generated before > > > > > > TaskManager > > > > > > > > > > > started. > > > > > > > > > > > > >> > > > > > > > > > > > > >> Override the calculated configurations through -D > > now > > > > > seems > > > > > > > > > better. > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> Best, > > > > > > > > > > > > >> > > > > > > > > > > > > >> Yang > > > > > > > > > > > > >> > > > > > > > > > > > > >> Xintong Song <[hidden email]> 于2019年9月2日周一 > > > > > > 上午11:39写道： > > > > > > > > > > > > >> > > > > > > > > > > > > >> > I just updated the FLIP wiki page [1], with the > > > > > following > > > > > > > > > changes: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > - Network memory uses JVM direct memory, and > is > > > > > > accounted > > > > > > > > > when > > > > > > > > > > > > >> setting > > > > > > > > > > > > >> > JVM max direct memory size parameter. > > > > > > > > > > > > >> > - Use dynamic configurations (`-Dkey=value`) > to > > > > pass > > > > > > > > > calculated > > > > > > > > > > > > >> memory > > > > > > > > > > > > >> > configs into TaskExecutors, instead of ENV > > > > variables. > > > > > > > > > > > > >> > - Remove 'supporting memory reservation' from > > the > > > > > scope > > > > > > > of > > > > > > > > > this > > > > > > > > > > > > FLIP. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > @till @stephan, please take another look see if > > > there > > > > > are > > > > > > > any > > > > > > > > > > other > > > > > > > > > > > > >> > concerns. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Thank you~ > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Xintong Song > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > [1] > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > On Mon, Sep 2, 2019 at 11:13 AM Xintong Song < > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Sorry for the late response. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > - Regarding the `TaskExecutorSpecifics` > naming, > > > > let's > > > > > > > > discuss > > > > > > > > > > the > > > > > > > > > > > > >> detail > > > > > > > > > > > > >> > > in PR. > > > > > > > > > > > > >> > > - Regarding passing parameters into the > > > > > `TaskExecutor`, > > > > > > +1 > > > > > > > > for > > > > > > > > > > > using > > > > > > > > > > > > >> > > dynamic configuration at the moment, given > that > > > > there > > > > > > are > > > > > > > > more > > > > > > > > > > > > >> questions > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > > be discussed to have a general framework for > > > > > overwriting > > > > > > > > > > > > >> configurations > > > > > > > > > > > > >> > > with ENV variables. > > > > > > > > > > > > >> > > - Regarding memory reservation, I double > checked > > > > with > > > > > Yu > > > > > > > and > > > > > > > > > he > > > > > > > > > > > will > > > > > > > > > > > > >> take > > > > > > > > > > > > >> > > care of it. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Thank you~ > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Xintong Song > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > On Thu, Aug 29, 2019 at 7:35 PM Till Rohrmann > < > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > >> What I forgot to add is that we could tackle > > > > > specifying > > > > > > > the > > > > > > > > > > > > >> > configuration > > > > > > > > > > > > >> > >> fully in an incremental way and that the full > > > > > > > specification > > > > > > > > > > > should > > > > > > > > > > > > be > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> desired end state. > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> On Thu, Aug 29, 2019 at 1:33 PM Till > Rohrmann < > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > I think our goal should be that the > > > configuration > > > > > is > > > > > > > > fully > > > > > > > > > > > > >> specified > > > > > > > > > > > > >> > >> when > > > > > > > > > > > > >> > >> > the process is started. By considering the > > > > internal > > > > > > > > > > calculation > > > > > > > > > > > > >> step > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > >> be > > > > > > > > > > > > >> > >> > rather validate existing values and > calculate > > > > > missing > > > > > > > > ones, > > > > > > > > > > > these > > > > > > > > > > > > >> two > > > > > > > > > > > > >> > >> > proposal shouldn't even conflict (given > > > > > determinism). > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > Since we don't want to change an existing > > > > > > > > flink-conf.yaml, > > > > > > > > > > > > >> specifying > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> > full configuration would require to pass in > > the > > > > > > options > > > > > > > > > > > > >> differently. > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > One way could be the ENV variables > approach. > > > The > > > > > > reason > > > > > > > > why > > > > > > > > > > I'm > > > > > > > > > > > > >> trying > > > > > > > > > > > > >> > >> to > > > > > > > > > > > > >> > >> > exclude this feature from the FLIP is that > I > > > > > believe > > > > > > it > > > > > > > > > > needs a > > > > > > > > > > > > bit > > > > > > > > > > > > >> > more > > > > > > > > > > > > >> > >> > discussion. Just some questions which come > to > > > my > > > > > > mind: > > > > > > > > What > > > > > > > > > > > would > > > > > > > > > > > > >> be > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> > exact format (FLINK_KEY_NAME)? Would we > > > support a > > > > > dot > > > > > > > > > > separator > > > > > > > > > > > > >> which > > > > > > > > > > > > >> > is > > > > > > > > > > > > >> > >> > supported by some systems (FLINK.KEY.NAME > )? > > If > > > > we > > > > > > > accept > > > > > > > > > the > > > > > > > > > > > dot > > > > > > > > > > > > >> > >> > separator what would be the order of > > precedence > > > > if > > > > > > > there > > > > > > > > > are > > > > > > > > > > > two > > > > > > > > > > > > >> ENV > > > > > > > > > > > > >> > >> > variables defined (FLINK_KEY_NAME and > > > > > FLINK.KEY.NAME > > > > > > )? > > > > > > > > > What > > > > > > > > > > is > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> > precedence of env variable vs. dynamic > > > > > configuration > > > > > > > > value > > > > > > > > > > > > >> specified > > > > > > > > > > > > >> > >> via -D? > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > Another approach could be to pass in the > > > dynamic > > > > > > > > > > configuration > > > > > > > > > > > > >> values > > > > > > > > > > > > >> > >> via > > > > > > > > > > > > >> > >> > `-Dkey=value` to the Flink process. For > that > > we > > > > > don't > > > > > > > > have > > > > > > > > > to > > > > > > > > > > > > >> change > > > > > > > > > > > > >> > >> > anything because the functionality already > > > > exists. > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > Cheers, > > > > > > > > > > > > >> > >> > Till > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > On Thu, Aug 29, 2019 at 12:50 PM Stephan > > Ewen < > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> >> I see. Under the assumption of strict > > > > determinism > > > > > > that > > > > > > > > > > should > > > > > > > > > > > > >> work. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> The original proposal had this point > "don't > > > > > compute > > > > > > > > inside > > > > > > > > > > the > > > > > > > > > > > > TM, > > > > > > > > > > > > >> > >> compute > > > > > > > > > > > > >> > >> >> outside and supply a full config", because > > > that > > > > > > > sounded > > > > > > > > > more > > > > > > > > > > > > >> > intuitive. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> On Thu, Aug 29, 2019 at 12:15 PM Till > > > Rohrmann < > > > > > > > > > > > > >> [hidden email] > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > >> >> wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> >> > My understanding was that before > starting > > > the > > > > > > Flink > > > > > > > > > > process > > > > > > > > > > > we > > > > > > > > > > > > >> > call a > > > > > > > > > > > > >> > >> >> > utility which calculates these values. I > > > > assume > > > > > > that > > > > > > > > > this > > > > > > > > > > > > >> utility > > > > > > > > > > > > >> > >> will > > > > > > > > > > > > >> > >> >> do > > > > > > > > > > > > >> > >> >> > the calculation based on a set of > > configured > > > > > > values > > > > > > > > > > (process > > > > > > > > > > > > >> > memory, > > > > > > > > > > > > >> > >> >> flink > > > > > > > > > > > > >> > >> >> > memory, network memory etc.). Assuming > > that > > > > > these > > > > > > > > values > > > > > > > > > > > don't > > > > > > > > > > > > >> > differ > > > > > > > > > > > > >> > >> >> from > > > > > > > > > > > > >> > >> >> > the values with which the JVM is > started, > > it > > > > > > should > > > > > > > be > > > > > > > > > > > > possible > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > >> >> > recompute them in the Flink process in > > order > > > > to > > > > > > set > > > > > > > > the > > > > > > > > > > > > values. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > On Thu, Aug 29, 2019 at 11:29 AM Stephan > > > Ewen > > > > < > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > When computing the values in the JVM > > > process > > > > > > after > > > > > > > > it > > > > > > > > > > > > started, > > > > > > > > > > > > >> > how > > > > > > > > > > > > >> > >> >> would > > > > > > > > > > > > >> > >> >> > > you deal with values like Max Direct > > > Memory, > > > > > > > > Metaspace > > > > > > > > > > > size. > > > > > > > > > > > > >> > native > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > reservation (reduce heap size), etc? > All > > > the > > > > > > > values > > > > > > > > > that > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> parameters > > > > > > > > > > > > >> > >> >> > to > > > > > > > > > > > > >> > >> >> > > the JVM process and that need to be > > > supplied > > > > > at > > > > > > > > > process > > > > > > > > > > > > >> startup? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > On Wed, Aug 28, 2019 at 4:46 PM Till > > > > Rohrmann > > > > > < > > > > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > Thanks for the clarification. I have > > > some > > > > > more > > > > > > > > > > comments: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > - I would actually split the logic > to > > > > > compute > > > > > > > the > > > > > > > > > > > process > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > > requirements and storing the values > > into > > > > two > > > > > > > > things. > > > > > > > > > > > E.g. > > > > > > > > > > > > >> one > > > > > > > > > > > > >> > >> could > > > > > > > > > > > > >> > >> >> > name > > > > > > > > > > > > >> > >> >> > > > the former > TaskExecutorProcessUtility > > > and > > > > > the > > > > > > > > > latter > > > > > > > > > > > > >> > >> >> > > > TaskExecutorProcessMemory. But we > can > > > > > discuss > > > > > > > this > > > > > > > > > on > > > > > > > > > > > the > > > > > > > > > > > > PR > > > > > > > > > > > > >> > >> since > > > > > > > > > > > > >> > >> >> it's > > > > > > > > > > > > >> > >> >> > > > just a naming detail. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > - Generally, I'm not opposed to > making > > > > > > > > configuration > > > > > > > > > > > > values > > > > > > > > > > > > >> > >> >> overridable > > > > > > > > > > > > >> > >> >> > > by > > > > > > > > > > > > >> > >> >> > > > ENV variables. I think this is a > very > > > good > > > > > > idea > > > > > > > > and > > > > > > > > > > > makes > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> >> > > > configurability of Flink processes > > > easier. > > > > > > > > However, > > > > > > > > > I > > > > > > > > > > > > think > > > > > > > > > > > > >> > that > > > > > > > > > > > > >> > >> >> adding > > > > > > > > > > > > >> > >> >> > > > this functionality should not be > part > > of > > > > > this > > > > > > > FLIP > > > > > > > > > > > because > > > > > > > > > > > > >> it > > > > > > > > > > > > >> > >> would > > > > > > > > > > > > >> > >> >> > > simply > > > > > > > > > > > > >> > >> >> > > > widen the scope unnecessarily. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > The reasons why I believe it is > > > > unnecessary > > > > > > are > > > > > > > > the > > > > > > > > > > > > >> following: > > > > > > > > > > > > >> > >> For > > > > > > > > > > > > >> > >> >> Yarn > > > > > > > > > > > > >> > >> >> > > we > > > > > > > > > > > > >> > >> >> > > > already create write a > flink-conf.yaml > > > > which > > > > > > > could > > > > > > > > > be > > > > > > > > > > > > >> populated > > > > > > > > > > > > >> > >> with > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > memory settings. For the other > > processes > > > > it > > > > > > > should > > > > > > > > > not > > > > > > > > > > > > make > > > > > > > > > > > > >> a > > > > > > > > > > > > >> > >> >> > difference > > > > > > > > > > > > >> > >> >> > > > whether the loaded Configuration is > > > > > populated > > > > > > > with > > > > > > > > > the > > > > > > > > > > > > >> memory > > > > > > > > > > > > >> > >> >> settings > > > > > > > > > > > > >> > >> >> > > from > > > > > > > > > > > > >> > >> >> > > > ENV variables or by using > > > > > > > > TaskExecutorProcessUtility > > > > > > > > > > to > > > > > > > > > > > > >> compute > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > missing > > > > > > > > > > > > >> > >> >> > > > values from the loaded > configuration. > > If > > > > the > > > > > > > > latter > > > > > > > > > > > would > > > > > > > > > > > > >> not > > > > > > > > > > > > >> > be > > > > > > > > > > > > >> > >> >> > possible > > > > > > > > > > > > >> > >> >> > > > (wrong or missing configuration > > values), > > > > > then > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > not > > > > > > > > > > > > >> > have > > > > > > > > > > > > >> > >> >> been > > > > > > > > > > > > >> > >> >> > > able > > > > > > > > > > > > >> > >> >> > > > to actually start the process in the > > > first > > > > > > > place. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > - Concerning the memory > reservation: I > > > > agree > > > > > > > with > > > > > > > > > you > > > > > > > > > > > that > > > > > > > > > > > > >> we > > > > > > > > > > > > >> > >> need > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > memory reservation functionality to > > make > > > > > > > streaming > > > > > > > > > > jobs > > > > > > > > > > > > work > > > > > > > > > > > > >> > with > > > > > > > > > > > > >> > >> >> > > "managed" > > > > > > > > > > > > >> > >> >> > > > memory. However, w/o this > > functionality > > > > the > > > > > > > whole > > > > > > > > > Flip > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> already > > > > > > > > > > > > >> > >> >> > > bring > > > > > > > > > > > > >> > >> >> > > > a good amount of improvements to our > > > users > > > > > > when > > > > > > > > > > running > > > > > > > > > > > > >> batch > > > > > > > > > > > > >> > >> jobs. > > > > > > > > > > > > >> > >> >> > > > Moreover, by keeping the scope > smaller > > > we > > > > > can > > > > > > > > > complete > > > > > > > > > > > the > > > > > > > > > > > > >> FLIP > > > > > > > > > > > > >> > >> >> faster. > > > > > > > > > > > > >> > >> >> > > > Hence, I would propose to address > the > > > > memory > > > > > > > > > > reservation > > > > > > > > > > > > >> > >> >> functionality > > > > > > > > > > > > >> > >> >> > > as a > > > > > > > > > > > > >> > >> >> > > > follow up FLIP (which Yu is working > on > > > if > > > > > I'm > > > > > > > not > > > > > > > > > > > > mistaken). > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > On Wed, Aug 28, 2019 at 11:43 AM > Yang > > > > Wang < > > > > > > > > > > > > >> > >> [hidden email]> > > > > > > > > > > > > >> > >> >> > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > Just add my 2 cents. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > Using environment variables to > > > override > > > > > the > > > > > > > > > > > > configuration > > > > > > > > > > > > >> for > > > > > > > > > > > > >> > >> >> > different > > > > > > > > > > > > >> > >> >> > > > > taskmanagers is better. > > > > > > > > > > > > >> > >> >> > > > > We do not need to generate > dedicated > > > > > > > > > flink-conf.yaml > > > > > > > > > > > for > > > > > > > > > > > > >> all > > > > > > > > > > > > >> > >> >> > > > taskmanagers. > > > > > > > > > > > > >> > >> >> > > > > A common flink-conf.yam and > > different > > > > > > > > environment > > > > > > > > > > > > >> variables > > > > > > > > > > > > >> > are > > > > > > > > > > > > >> > >> >> > enough. > > > > > > > > > > > > >> > >> >> > > > > By reducing the distributed cached > > > > files, > > > > > it > > > > > > > > could > > > > > > > > > > > make > > > > > > > > > > > > >> > >> launching > > > > > > > > > > > > >> > >> >> a > > > > > > > > > > > > >> > >> >> > > > > taskmanager faster. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > Stephan gives a good suggestion > that > > > we > > > > > > could > > > > > > > > move > > > > > > > > > > the > > > > > > > > > > > > >> logic > > > > > > > > > > > > >> > >> into > > > > > > > > > > > > >> > >> >> > > > > "GlobalConfiguration.loadConfig()" > > > > method. > > > > > > > > > > > > >> > >> >> > > > > Maybe the client could also > benefit > > > from > > > > > > this. > > > > > > > > > > > Different > > > > > > > > > > > > >> > users > > > > > > > > > > > > >> > >> do > > > > > > > > > > > > >> > >> >> not > > > > > > > > > > > > >> > >> >> > > > have > > > > > > > > > > > > >> > >> >> > > > > to export FLINK_CONF_DIR to update > > few > > > > > > config > > > > > > > > > > options. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > Best, > > > > > > > > > > > > >> > >> >> > > > > Yang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > Stephan Ewen <[hidden email]> > > > > > > 于2019年8月28日周三 > > > > > > > > > > > 上午1:21写道： > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > One note on the Environment > > > Variables > > > > > and > > > > > > > > > > > > Configuration > > > > > > > > > > > > >> > >> >> discussion. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > My understanding is that passed > > ENV > > > > > > > variables > > > > > > > > > are > > > > > > > > > > > > added > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > > > > configuration in the > > > > > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > > > > >> > >> method > > > > > > > > > > > > >> > >> >> (or > > > > > > > > > > > > >> > >> >> > > > > > similar). > > > > > > > > > > > > >> > >> >> > > > > > For all the code inside Flink, > it > > > > looks > > > > > > like > > > > > > > > the > > > > > > > > > > > data > > > > > > > > > > > > >> was > > > > > > > > > > > > >> > in > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > config > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > start with, just that the > scripts > > > that > > > > > > > compute > > > > > > > > > the > > > > > > > > > > > > >> > variables > > > > > > > > > > > > >> > >> can > > > > > > > > > > > > >> > >> >> > pass > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > values to the process without > > > actually > > > > > > > needing > > > > > > > > > to > > > > > > > > > > > > write > > > > > > > > > > > > >> a > > > > > > > > > > > > >> > >> file. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > For example the > > > > > > > > > "GlobalConfiguration.loadConfig()" > > > > > > > > > > > > >> method > > > > > > > > > > > > >> > >> would > > > > > > > > > > > > >> > >> >> > take > > > > > > > > > > > > >> > >> >> > > > any > > > > > > > > > > > > >> > >> >> > > > > > ENV variable prefixed with > "flink" > > > and > > > > > add > > > > > > > it > > > > > > > > > as a > > > > > > > > > > > > >> config > > > > > > > > > > > > >> > >> key. > > > > > > > > > > > > >> > >> >> > > > > > > "flink_taskmanager_memory_size=2g" > > > > would > > > > > > > > become > > > > > > > > > > > > >> > >> >> > > > "taskmanager.memory.size: > > > > > > > > > > > > >> > >> >> > > > > > 2g". > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > On Tue, Aug 27, 2019 at 4:05 PM > > > > Xintong > > > > > > > Song < > > > > > > > > > > > > >> > >> >> > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thanks for the comments, Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > I've also seen your comments > on > > > the > > > > > wiki > > > > > > > > page, > > > > > > > > > > but > > > > > > > > > > > > >> let's > > > > > > > > > > > > >> > >> keep > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > > > > discussion here. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding > > > 'TaskExecutorSpecifics', > > > > > how > > > > > > > do > > > > > > > > > you > > > > > > > > > > > > think > > > > > > > > > > > > >> > about > > > > > > > > > > > > >> > >> >> > naming > > > > > > > > > > > > >> > >> >> > > it > > > > > > > > > > > > >> > >> >> > > > > > > > 'TaskExecutorResourceSpecifics'. > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding passing memory > > > > > > configurations > > > > > > > > into > > > > > > > > > > > task > > > > > > > > > > > > >> > >> executors, > > > > > > > > > > > > >> > >> >> > I'm > > > > > > > > > > > > >> > >> >> > > in > > > > > > > > > > > > >> > >> >> > > > > > favor > > > > > > > > > > > > >> > >> >> > > > > > > of do it via environment > > variables > > > > > > rather > > > > > > > > than > > > > > > > > > > > > >> > >> configurations, > > > > > > > > > > > > >> > >> >> > with > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > following two reasons. > > > > > > > > > > > > >> > >> >> > > > > > > - It is easier to keep the > > > memory > > > > > > > options > > > > > > > > > once > > > > > > > > > > > > >> > calculate > > > > > > > > > > > > >> > >> >> not to > > > > > > > > > > > > >> > >> >> > > be > > > > > > > > > > > > >> > >> >> > > > > > > changed with environment > > variables > > > > > > rather > > > > > > > > than > > > > > > > > > > > > >> > >> configurations. > > > > > > > > > > > > >> > >> >> > > > > > > - I'm not sure whether we > > should > > > > > write > > > > > > > the > > > > > > > > > > > > >> > configuration > > > > > > > > > > > > >> > >> in > > > > > > > > > > > > >> > >> >> > > startup > > > > > > > > > > > > >> > >> >> > > > > > > scripts. Writing changes into > > the > > > > > > > > > configuration > > > > > > > > > > > > files > > > > > > > > > > > > >> > when > > > > > > > > > > > > >> > >> >> > running > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > startup scripts does not > sounds > > > > right > > > > > to > > > > > > > me. > > > > > > > > > Or > > > > > > > > > > we > > > > > > > > > > > > >> could > > > > > > > > > > > > >> > >> make > > > > > > > > > > > > >> > >> >> a > > > > > > > > > > > > >> > >> >> > > copy > > > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > > > >> > >> >> > > > > > > configuration files per flink > > > > cluster, > > > > > > and > > > > > > > > > make > > > > > > > > > > > the > > > > > > > > > > > > >> task > > > > > > > > > > > > >> > >> >> executor > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > load > > > > > > > > > > > > >> > >> >> > > > > > > from the copy, and clean up > the > > > copy > > > > > > after > > > > > > > > the > > > > > > > > > > > > >> cluster is > > > > > > > > > > > > >> > >> >> > shutdown, > > > > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > is complicated. (I think this > is > > > > also > > > > > > what > > > > > > > > > > Stephan > > > > > > > > > > > > >> means > > > > > > > > > > > > >> > in > > > > > > > > > > > > >> > >> >> his > > > > > > > > > > > > >> > >> >> > > > comment > > > > > > > > > > > > >> > >> >> > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > the wiki page?) > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding reserving memory, > I > > > > think > > > > > > this > > > > > > > > > > change > > > > > > > > > > > > >> should > > > > > > > > > > > > >> > be > > > > > > > > > > > > >> > >> >> > > included > > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > > >> > >> >> > > > > > > this FLIP. I think a big part > of > > > > > > > motivations > > > > > > > > > of > > > > > > > > > > > this > > > > > > > > > > > > >> FLIP > > > > > > > > > > > > >> > >> is > > > > > > > > > > > > >> > >> >> to > > > > > > > > > > > > >> > >> >> > > unify > > > > > > > > > > > > >> > >> >> > > > > > > memory configuration for > > > streaming / > > > > > > batch > > > > > > > > and > > > > > > > > > > > make > > > > > > > > > > > > it > > > > > > > > > > > > >> > easy > > > > > > > > > > > > >> > >> >> for > > > > > > > > > > > > >> > >> >> > > > > > configuring > > > > > > > > > > > > >> > >> >> > > > > > > rocksdb memory. If we don't > > > support > > > > > > memory > > > > > > > > > > > > >> reservation, > > > > > > > > > > > > >> > >> then > > > > > > > > > > > > >> > >> >> > > > streaming > > > > > > > > > > > > >> > >> >> > > > > > jobs > > > > > > > > > > > > >> > >> >> > > > > > > cannot use managed memory > > (neither > > > > > > on-heap > > > > > > > > or > > > > > > > > > > > > >> off-heap), > > > > > > > > > > > > >> > >> which > > > > > > > > > > > > >> > >> >> > > makes > > > > > > > > > > > > >> > >> >> > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > FLIP incomplete. > > > > > > > > > > > > >> > >> >> > > > > > > - Regarding network memory, I > > > think > > > > > you > > > > > > > are > > > > > > > > > > > right. I > > > > > > > > > > > > >> > think > > > > > > > > > > > > >> > >> we > > > > > > > > > > > > >> > >> >> > > > probably > > > > > > > > > > > > >> > >> >> > > > > > > don't need to change network > > stack > > > > > from > > > > > > > > using > > > > > > > > > > > direct > > > > > > > > > > > > >> > >> memory to > > > > > > > > > > > > >> > >> >> > > using > > > > > > > > > > > > >> > >> >> > > > > > unsafe > > > > > > > > > > > > >> > >> >> > > > > > > native memory. Network memory > > size > > > > is > > > > > > > > > > > deterministic, > > > > > > > > > > > > >> > >> cannot be > > > > > > > > > > > > >> > >> >> > > > reserved > > > > > > > > > > > > >> > >> >> > > > > > as > > > > > > > > > > > > >> > >> >> > > > > > > managed memory does, and > cannot > > be > > > > > > > > overused. I > > > > > > > > > > > think > > > > > > > > > > > > >> it > > > > > > > > > > > > >> > >> also > > > > > > > > > > > > >> > >> >> > works > > > > > > > > > > > > >> > >> >> > > if > > > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > simply keep using direct > memory > > > for > > > > > > > network > > > > > > > > > and > > > > > > > > > > > > >> include > > > > > > > > > > > > >> > it > > > > > > > > > > > > >> > >> in > > > > > > > > > > > > >> > >> >> jvm > > > > > > > > > > > > >> > >> >> > > max > > > > > > > > > > > > >> > >> >> > > > > > > direct memory size. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > Xintong Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > On Tue, Aug 27, 2019 at 8:12 > PM > > > Till > > > > > > > > Rohrmann > > > > > > > > > < > > > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Hi Xintong, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > thanks for addressing the > > > comments > > > > > and > > > > > > > > > adding > > > > > > > > > > a > > > > > > > > > > > > more > > > > > > > > > > > > >> > >> >> detailed > > > > > > > > > > > > >> > >> >> > > > > > > > implementation plan. I have > a > > > > couple > > > > > > of > > > > > > > > > > comments > > > > > > > > > > > > >> > >> concerning > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > > > > > implementation plan: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > - The name > > > `TaskExecutorSpecifics` > > > > > is > > > > > > > not > > > > > > > > > > really > > > > > > > > > > > > >> > >> >> descriptive. > > > > > > > > > > > > >> > >> >> > > > > Choosing > > > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > different name could help > > here. > > > > > > > > > > > > >> > >> >> > > > > > > > - I'm not sure whether I > would > > > > pass > > > > > > the > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> configuration to > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > > > TaskExecutor via environment > > > > > > variables. > > > > > > > I > > > > > > > > > > think > > > > > > > > > > > it > > > > > > > > > > > > >> > would > > > > > > > > > > > > >> > >> be > > > > > > > > > > > > >> > >> >> > > better > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > write > > > > > > > > > > > > >> > >> >> > > > > > > > it into the configuration > one > > > uses > > > > > to > > > > > > > > start > > > > > > > > > > the > > > > > > > > > > > TM > > > > > > > > > > > > >> > >> process. > > > > > > > > > > > > >> > >> >> > > > > > > > - If possible, I would > exclude > > > the > > > > > > > memory > > > > > > > > > > > > >> reservation > > > > > > > > > > > > >> > >> from > > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > add this as part of a > > dedicated > > > > > FLIP. > > > > > > > > > > > > >> > >> >> > > > > > > > - If possible, then I would > > > > exclude > > > > > > > > changes > > > > > > > > > to > > > > > > > > > > > the > > > > > > > > > > > > >> > >> network > > > > > > > > > > > > >> > >> >> > stack > > > > > > > > > > > > >> > >> >> > > > from > > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > FLIP. Maybe we can simply > say > > > that > > > > > the > > > > > > > > > direct > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> needed > > > > > > > > > > > > >> > >> >> by > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > > network > > > > > > > > > > > > >> > >> >> > > > > > > > stack is the framework > direct > > > > memory > > > > > > > > > > > requirement. > > > > > > > > > > > > >> > >> Changing > > > > > > > > > > > > >> > >> >> how > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > is allocated can happen in a > > > > second > > > > > > > step. > > > > > > > > > This > > > > > > > > > > > > would > > > > > > > > > > > > >> > keep > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > scope > > > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > FLIP smaller. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > On Thu, Aug 22, 2019 at 2:51 > > PM > > > > > > Xintong > > > > > > > > > Song < > > > > > > > > > > > > >> > >> >> > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Hi everyone, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > I just updated the FLIP > > > document > > > > > on > > > > > > > wiki > > > > > > > > > > [1], > > > > > > > > > > > > with > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > > following > > > > > > > > > > > > >> > >> >> > > > > > > changes. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > - Removed open question > > > > > regarding > > > > > > > > > > > > MemorySegment > > > > > > > > > > > > >> > >> >> > allocation. > > > > > > > > > > > > >> > >> >> > > As > > > > > > > > > > > > >> > >> >> > > > > > > > > discussed, we exclude > > this > > > > > topic > > > > > > > from > > > > > > > > > the > > > > > > > > > > > > >> scope of > > > > > > > > > > > > >> > >> this > > > > > > > > > > > > >> > >> >> > > FLIP. > > > > > > > > > > > > >> > >> >> > > > > > > > > - Updated content about > > JVM > > > > > > direct > > > > > > > > > memory > > > > > > > > > > > > >> > parameter > > > > > > > > > > > > >> > >> >> > > according > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > recent > > > > > > > > > > > > >> > >> >> > > > > > > > > discussions, and moved > > the > > > > > other > > > > > > > > > options > > > > > > > > > > to > > > > > > > > > > > > >> > >> "Rejected > > > > > > > > > > > > >> > >> >> > > > > > Alternatives" > > > > > > > > > > > > >> > >> >> > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > moment. > > > > > > > > > > > > >> > >> >> > > > > > > > > - Added implementation > > > steps. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > Xintong Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > [1] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > On Mon, Aug 19, 2019 at > 7:16 > > > PM > > > > > > > Stephan > > > > > > > > > > Ewen < > > > > > > > > > > > > >> > >> >> > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong: Concerning > "wait > > > for > > > > > > > memory > > > > > > > > > > users > > > > > > > > > > > > >> before > > > > > > > > > > > > >> > >> task > > > > > > > > > > > > >> > >> >> > > dispose > > > > > > > > > > > > >> > >> >> > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > release": I agree, > that's > > > how > > > > it > > > > > > > > should > > > > > > > > > > be. > > > > > > > > > > > > >> Let's > > > > > > > > > > > > >> > >> try it > > > > > > > > > > > > >> > >> >> > out. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Xintong @Jingsong: > > > > Concerning " > > > > > > JVM > > > > > > > > > does > > > > > > > > > > > not > > > > > > > > > > > > >> wait > > > > > > > > > > > > >> > >> for > > > > > > > > > > > > >> > >> >> GC > > > > > > > > > > > > >> > >> >> > > when > > > > > > > > > > > > >> > >> >> > > > > > > > allocating > > > > > > > > > > > > >> > >> >> > > > > > > > > > direct memory buffer": > > There > > > > > seems > > > > > > > to > > > > > > > > be > > > > > > > > > > > > pretty > > > > > > > > > > > > >> > >> >> elaborate > > > > > > > > > > > > >> > >> >> > > logic > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > free > > > > > > > > > > > > >> > >> >> > > > > > > > > > buffers when allocating > > new > > > > > ones. > > > > > > > See > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/tip/src/share/classes/java/nio/Bits.java#l643 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > @Till: Maybe. If we > assume > > > > that > > > > > > the > > > > > > > > JVM > > > > > > > > > > > > default > > > > > > > > > > > > >> > works > > > > > > > > > > > > >> > >> >> (like > > > > > > > > > > > > >> > >> >> > > > going > > > > > > > > > > > > >> > >> >> > > > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > option 2 and not setting > > > > > > > > > > > > >> "-XX:MaxDirectMemorySize" > > > > > > > > > > > > >> > at > > > > > > > > > > > > >> > >> >> all), > > > > > > > > > > > > >> > >> >> > > > then > > > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > > > >> > >> >> > > > > > > > think > > > > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > > should be okay to set > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > > >> to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > "off_heap_managed_memory + > > > > > > > > > direct_memory" > > > > > > > > > > > even > > > > > > > > > > > > >> if > > > > > > > > > > > > >> > we > > > > > > > > > > > > >> > >> use > > > > > > > > > > > > >> > >> >> > > > RocksDB. > > > > > > > > > > > > >> > >> >> > > > > > > That > > > > > > > > > > > > >> > >> >> > > > > > > > > is a > > > > > > > > > > > > >> > >> >> > > > > > > > > > big if, though, I > honestly > > > > have > > > > > no > > > > > > > > idea > > > > > > > > > :D > > > > > > > > > > > > >> Would be > > > > > > > > > > > > >> > >> >> good to > > > > > > > > > > > > >> > >> >> > > > > > > understand > > > > > > > > > > > > >> > >> >> > > > > > > > > > this, though, because > this > > > > would > > > > > > > > affect > > > > > > > > > > > option > > > > > > > > > > > > >> (2) > > > > > > > > > > > > >> > >> and > > > > > > > > > > > > >> > >> >> > option > > > > > > > > > > > > >> > >> >> > > > > > (1.2). > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > On Mon, Aug 19, 2019 at > > 4:44 > > > > PM > > > > > > > > Xintong > > > > > > > > > > > Song < > > > > > > > > > > > > >> > >> >> > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thanks for the inputs, > > > > > Jingsong. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Let me try to > summarize > > > your > > > > > > > points. > > > > > > > > > > > Please > > > > > > > > > > > > >> > correct > > > > > > > > > > > > >> > >> >> me if > > > > > > > > > > > > >> > >> >> > > I'm > > > > > > > > > > > > >> > >> >> > > > > > > wrong. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - Memory consumers > > > should > > > > > > > always > > > > > > > > > > avoid > > > > > > > > > > > > >> > returning > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > > > > segments > > > > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > memory manager > while > > > > there > > > > > > are > > > > > > > > > still > > > > > > > > > > > > >> > un-cleaned > > > > > > > > > > > > >> > >> >> > > > structures / > > > > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > may use the memory. > > > > > > Otherwise, > > > > > > > it > > > > > > > > > > would > > > > > > > > > > > > >> cause > > > > > > > > > > > > >> > >> >> serious > > > > > > > > > > > > >> > >> >> > > > > problems > > > > > > > > > > > > >> > >> >> > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > > > >> > >> >> > > > > > > > > > > multiple consumers > > > trying > > > > > to > > > > > > > use > > > > > > > > > the > > > > > > > > > > > same > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > segment. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > - JVM does not wait > > for > > > > GC > > > > > > when > > > > > > > > > > > > allocating > > > > > > > > > > > > >> > >> direct > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > > > > buffer. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Therefore even we > set > > > > > proper > > > > > > > max > > > > > > > > > > direct > > > > > > > > > > > > >> memory > > > > > > > > > > > > >> > >> size > > > > > > > > > > > > >> > >> >> > > limit, > > > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > > > > still > > > > > > > > > > > > >> > >> >> > > > > > > > > > > encounter direct > > memory > > > > oom > > > > > > if > > > > > > > > the > > > > > > > > > GC > > > > > > > > > > > > >> cleaning > > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > > >> > >> >> > > > slower > > > > > > > > > > > > >> > >> >> > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct memory > > > allocation. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Am I understanding > this > > > > > > correctly? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Xintong Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > On Mon, Aug 19, 2019 > at > > > 4:21 > > > > > PM > > > > > > > > > > > JingsongLee > > > > > > > > > > > > < > > > > > > > > > > > > >> > >> >> > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > .invalid> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Hi stephan: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About option 2: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > if additional > threads > > > not > > > > > > > cleanly > > > > > > > > > shut > > > > > > > > > > > > down > > > > > > > > > > > > >> > >> before > > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > > >> > >> >> > can > > > > > > > > > > > > >> > >> >> > > > > exit > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > task: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > In the current case > of > > > > > memory > > > > > > > > reuse, > > > > > > > > > > it > > > > > > > > > > > > has > > > > > > > > > > > > >> > >> freed up > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > uses. If this > memory > > is > > > > > used > > > > > > by > > > > > > > > > other > > > > > > > > > > > > tasks > > > > > > > > > > > > >> > and > > > > > > > > > > > > >> > >> >> > > > asynchronous > > > > > > > > > > > > >> > >> >> > > > > > > > threads > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of exited task may > > > still > > > > be > > > > > > > > > writing, > > > > > > > > > > > > there > > > > > > > > > > > > >> > will > > > > > > > > > > > > >> > >> be > > > > > > > > > > > > >> > >> >> > > > > concurrent > > > > > > > > > > > > >> > >> >> > > > > > > > > security > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > problems, and even > > lead > > > > to > > > > > > > errors > > > > > > > > > in > > > > > > > > > > > user > > > > > > > > > > > > >> > >> computing > > > > > > > > > > > > >> > >> >> > > > results. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > So I think this is a > > > > serious > > > > > > and > > > > > > > > > > > > intolerable > > > > > > > > > > > > >> > >> bug, No > > > > > > > > > > > > >> > >> >> > > matter > > > > > > > > > > > > >> > >> >> > > > > > what > > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > option is, it > should > > be > > > > > > > avoided. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > About direct memory > > > > cleaned > > > > > by > > > > > > > GC: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > I don't think it is > a > > > good > > > > > > idea, > > > > > > > > > I've > > > > > > > > > > > > >> > >> encountered so > > > > > > > > > > > > >> > >> >> > many > > > > > > > > > > > > >> > >> >> > > > > > > > situations > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > that it's too late > > for > > > GC > > > > > to > > > > > > > > cause > > > > > > > > > > > > >> > DirectMemory > > > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > > > >> > >> >> > > > Release > > > > > > > > > > > > >> > >> >> > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > allocate > DirectMemory > > > > > depend > > > > > > on > > > > > > > > the > > > > > > > > > > > type > > > > > > > > > > > > of > > > > > > > > > > > > >> > user > > > > > > > > > > > > >> > >> >> job, > > > > > > > > > > > > >> > >> >> > > > which > > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > often beyond our > > > control. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Best, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Jingsong Lee > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > From:Stephan Ewen < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Send > > > Time:2019年8月19日(星期一) > > > > > > 15:56 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > To:dev < > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Subject:Re: > [DISCUSS] > > > > > FLIP-49: > > > > > > > > > Unified > > > > > > > > > > > > >> Memory > > > > > > > > > > > > >> > >> >> > > Configuration > > > > > > > > > > > > >> > >> >> > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > TaskExecutors > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > My main concern with > > > > option > > > > > 2 > > > > > > > > > > (manually > > > > > > > > > > > > >> release > > > > > > > > > > > > >> > >> >> memory) > > > > > > > > > > > > >> > >> >> > > is > > > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > segfaults > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > in the JVM send off > > all > > > > > sorts > > > > > > of > > > > > > > > > > alarms > > > > > > > > > > > on > > > > > > > > > > > > >> user > > > > > > > > > > > > >> > >> >> ends. > > > > > > > > > > > > >> > >> >> > So > > > > > > > > > > > > >> > >> >> > > we > > > > > > > > > > > > >> > >> >> > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > guarantee that this > > > never > > > > > > > happens. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > The trickyness is in > > > tasks > > > > > > that > > > > > > > > uses > > > > > > > > > > > data > > > > > > > > > > > > >> > >> >> structures / > > > > > > > > > > > > >> > >> >> > > > > > algorithms > > > > > > > > > > > > >> > >> >> > > > > > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > additional threads, > > like > > > > > hash > > > > > > > > table > > > > > > > > > > > > >> spill/read > > > > > > > > > > > > >> > >> and > > > > > > > > > > > > >> > >> >> > > sorting > > > > > > > > > > > > >> > >> >> > > > > > > threads. > > > > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > to ensure that these > > > > cleanly > > > > > > > shut > > > > > > > > > down > > > > > > > > > > > > >> before > > > > > > > > > > > > >> > we > > > > > > > > > > > > >> > >> can > > > > > > > > > > > > >> > >> >> > exit > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > task. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > I am not sure that > we > > > have > > > > > > that > > > > > > > > > > > guaranteed > > > > > > > > > > > > >> > >> already, > > > > > > > > > > > > >> > >> >> > > that's > > > > > > > > > > > > >> > >> >> > > > > why > > > > > > > > > > > > >> > >> >> > > > > > > > option > > > > > > > > > > > > >> > >> >> > > > > > > > > > 1.1 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > seemed simpler to > me. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > On Mon, Aug 19, 2019 > > at > > > > 3:42 > > > > > > PM > > > > > > > > > > Xintong > > > > > > > > > > > > >> Song < > > > > > > > > > > > > >> > >> >> > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thanks for the > > > comments, > > > > > > > > Stephan. > > > > > > > > > > > > >> Summarized > > > > > > > > > > > > >> > in > > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > > >> > >> >> > > way > > > > > > > > > > > > >> > >> >> > > > > > really > > > > > > > > > > > > >> > >> >> > > > > > > > > makes > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > things easier to > > > > > understand. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > I'm in favor of > > option > > > > 2, > > > > > at > > > > > > > > least > > > > > > > > > > for > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> moment. I > > > > > > > > > > > > >> > >> >> > > > think > > > > > > > > > > > > >> > >> >> > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > difficult to keep > it > > > > > > segfault > > > > > > > > safe > > > > > > > > > > for > > > > > > > > > > > > >> memory > > > > > > > > > > > > >> > >> >> > manager, > > > > > > > > > > > > >> > >> >> > > as > > > > > > > > > > > > >> > >> >> > > > > > long > > > > > > > > > > > > >> > >> >> > > > > > > as > > > > > > > > > > > > >> > >> >> > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > always > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > de-allocate the > > memory > > > > > > segment > > > > > > > > > when > > > > > > > > > > it > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> released > > > > > > > > > > > > >> > >> >> > from > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > consumers. Only if > > the > > > > > > memory > > > > > > > > > > consumer > > > > > > > > > > > > >> > continue > > > > > > > > > > > > >> > >> >> using > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > > buffer > > > > > > > > > > > > >> > >> >> > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > segment after > > > releasing > > > > > it, > > > > > > in > > > > > > > > > which > > > > > > > > > > > > case > > > > > > > > > > > > >> we > > > > > > > > > > > > >> > do > > > > > > > > > > > > >> > >> >> want > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > job > > > > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > fail > > > > > > > > > > > > >> > >> >> > > > > > > > > > so > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > detect the memory > > leak > > > > > > early. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.2, I > > > don't > > > > > > think > > > > > > > > this > > > > > > > > > > is > > > > > > > > > > > a > > > > > > > > > > > > >> good > > > > > > > > > > > > >> > >> idea. > > > > > > > > > > > > >> > >> >> > Not > > > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > > > >> > >> >> > > > > > > > because > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > assumption > (regular > > GC > > > > is > > > > > > > enough > > > > > > > > > to > > > > > > > > > > > > clean > > > > > > > > > > > > >> > >> direct > > > > > > > > > > > > >> > >> >> > > buffers) > > > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > always > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > true, but also it > > > makes > > > > > > harder > > > > > > > > for > > > > > > > > > > > > finding > > > > > > > > > > > > >> > >> >> problems > > > > > > > > > > > > >> > >> >> > in > > > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > > > >> > >> >> > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > overuse. E.g., > user > > > > > > configured > > > > > > > > > some > > > > > > > > > > > > direct > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> for > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > libraries. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > If the library > > > actually > > > > > use > > > > > > > more > > > > > > > > > > > direct > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> then > > > > > > > > > > > > >> > >> >> > > > > > configured, > > > > > > > > > > > > >> > >> >> > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cannot be cleaned > by > > > GC > > > > > > > because > > > > > > > > > they > > > > > > > > > > > are > > > > > > > > > > > > >> > still > > > > > > > > > > > > >> > >> in > > > > > > > > > > > > >> > >> >> > use, > > > > > > > > > > > > >> > >> >> > > > may > > > > > > > > > > > > >> > >> >> > > > > > lead > > > > > > > > > > > > >> > >> >> > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > overuse > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > of the total > > container > > > > > > memory. > > > > > > > > In > > > > > > > > > > that > > > > > > > > > > > > >> case, > > > > > > > > > > > > >> > >> if it > > > > > > > > > > > > >> > >> >> > > didn't > > > > > > > > > > > > >> > >> >> > > > > > touch > > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > default max direct > > > > memory > > > > > > > limit, > > > > > > > > > we > > > > > > > > > > > > cannot > > > > > > > > > > > > >> > get > > > > > > > > > > > > >> > >> a > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > OOM > > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > will become super > > hard > > > > to > > > > > > > > > understand > > > > > > > > > > > > which > > > > > > > > > > > > >> > >> part of > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to be updated. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > For option 1.1, it > > has > > > > the > > > > > > > > similar > > > > > > > > > > > > >> problem as > > > > > > > > > > > > >> > >> >> 1.2, if > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > > > exceeded > > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > memory does not > > reach > > > > the > > > > > > max > > > > > > > > > direct > > > > > > > > > > > > >> memory > > > > > > > > > > > > >> > >> limit > > > > > > > > > > > > >> > >> >> > > > specified > > > > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > dedicated > > parameter. I > > > > > think > > > > > > > it > > > > > > > > is > > > > > > > > > > > > >> slightly > > > > > > > > > > > > >> > >> better > > > > > > > > > > > > >> > >> >> > than > > > > > > > > > > > > >> > >> >> > > > > 1.2, > > > > > > > > > > > > >> > >> >> > > > > > > only > > > > > > > > > > > > >> > >> >> > > > > > > > > > > because > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we can tune the > > > > parameter. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > On Mon, Aug 19, > 2019 > > > at > > > > > 2:53 > > > > > > > PM > > > > > > > > > > > Stephan > > > > > > > > > > > > >> Ewen > > > > > > > > > > > > >> > < > > > > > > > > > > > > >> > >> >> > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > About the > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > > >> > >> discussion, > > > > > > > > > > > > >> > >> >> > maybe > > > > > > > > > > > > >> > >> >> > > > let > > > > > > > > > > > > >> > >> >> > > > > > me > > > > > > > > > > > > >> > >> >> > > > > > > > > > > summarize > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > it a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > bit differently: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > We have the > > > following > > > > > two > > > > > > > > > options: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (1) We let > > > > > MemorySegments > > > > > > be > > > > > > > > > > > > >> de-allocated > > > > > > > > > > > > >> > by > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > GC. > > > > > > > > > > > > >> > >> >> > > > That > > > > > > > > > > > > >> > >> >> > > > > > > makes > > > > > > > > > > > > >> > >> >> > > > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > segfault > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > safe. But then > we > > > > need a > > > > > > way > > > > > > > > to > > > > > > > > > > > > trigger > > > > > > > > > > > > >> GC > > > > > > > > > > > > >> > in > > > > > > > > > > > > >> > >> >> case > > > > > > > > > > > > >> > >> >> > > > > > > > de-allocation > > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > re-allocation > of a > > > > bunch > > > > > > of > > > > > > > > > > segments > > > > > > > > > > > > >> > happens > > > > > > > > > > > > >> > >> >> > quickly, > > > > > > > > > > > > >> > >> >> > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > often > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > case during > batch > > > > > > scheduling > > > > > > > > or > > > > > > > > > > task > > > > > > > > > > > > >> > restart. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - The > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > > >> (option > > > > > > > > > > > > >> > >> 1.1) > > > > > > > > > > > > >> > >> >> is > > > > > > > > > > > > >> > >> >> > one > > > > > > > > > > > > >> > >> >> > > > way > > > > > > > > > > > > >> > >> >> > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Another way > > > could > > > > be > > > > > > to > > > > > > > > > have a > > > > > > > > > > > > >> > dedicated > > > > > > > > > > > > >> > >> >> > > > bookkeeping > > > > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > MemoryManager > > > (option > > > > > > 1.2), > > > > > > > so > > > > > > > > > > that > > > > > > > > > > > > this > > > > > > > > > > > > >> > is a > > > > > > > > > > > > >> > >> >> > number > > > > > > > > > > > > >> > >> >> > > > > > > > independent > > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > parameter. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (2) We manually > > > > allocate > > > > > > and > > > > > > > > > > > > de-allocate > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > > >> > >> >> > > for > > > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > MemorySegments > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > (option 2). That > > way > > > > we > > > > > > need > > > > > > > > not > > > > > > > > > > > worry > > > > > > > > > > > > >> > about > > > > > > > > > > > > >> > >> >> > > triggering > > > > > > > > > > > > >> > >> >> > > > > GC > > > > > > > > > > > > >> > >> >> > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > some > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > threshold or > > > > > bookkeeping, > > > > > > > but > > > > > > > > it > > > > > > > > > > is > > > > > > > > > > > > >> harder > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > >> >> > prevent > > > > > > > > > > > > >> > >> >> > > > > > > > segfaults. > > > > > > > > > > > > >> > >> >> > > > > > > > > We > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be very careful > > > about > > > > > when > > > > > > > we > > > > > > > > > > > release > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > > > segments > > > > > > > > > > > > >> > >> >> > > > > > > (only > > > > > > > > > > > > >> > >> >> > > > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cleanup phase of > > the > > > > > main > > > > > > > > > thread). > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with > > option > > > > > 1.1, > > > > > > we > > > > > > > > > > > probably > > > > > > > > > > > > >> need > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > >> >> set > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > to > > > > > > > > > > > > >> > >> >> > > "off_heap_managed_memory + > > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct_memory" > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > have > > "direct_memory" > > > > as > > > > > a > > > > > > > > > separate > > > > > > > > > > > > >> reserved > > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > > >> > >> >> > > > pool. > > > > > > > > > > > > >> > >> >> > > > > > > > Because > > > > > > > > > > > > >> > >> >> > > > > > > > > if > > > > > > > > > > > > >> > >> >> > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > just > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > set > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > "off_heap_managed_memory + > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > jvm_overhead", > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > then there will > be > > > > times > > > > > > > when > > > > > > > > > that > > > > > > > > > > > > >> entire > > > > > > > > > > > > >> > >> >> memory is > > > > > > > > > > > > >> > >> >> > > > > > allocated > > > > > > > > > > > > >> > >> >> > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > buffers and we > > have > > > > > > nothing > > > > > > > > left > > > > > > > > > > for > > > > > > > > > > > > the > > > > > > > > > > > > >> > JVM > > > > > > > > > > > > >> > >> >> > > overhead. > > > > > > > > > > > > >> > >> >> > > > So > > > > > > > > > > > > >> > >> >> > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > either > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > way to > compensate > > > for > > > > > that > > > > > > > > > (again > > > > > > > > > > > some > > > > > > > > > > > > >> > safety > > > > > > > > > > > > >> > >> >> > margin > > > > > > > > > > > > >> > >> >> > > > > cutoff > > > > > > > > > > > > >> > >> >> > > > > > > > > value) > > > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > will exceed > > > container > > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > If we go with > > option > > > > > 1.2, > > > > > > we > > > > > > > > > need > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > >> > aware > > > > > > > > > > > > >> > >> >> that > > > > > > > > > > > > >> > >> >> > it > > > > > > > > > > > > >> > >> >> > > > > takes > > > > > > > > > > > > >> > >> >> > > > > > > > > > elaborate > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > logic > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > to push > recycling > > of > > > > > > direct > > > > > > > > > > buffers > > > > > > > > > > > > >> without > > > > > > > > > > > > >> > >> >> always > > > > > > > > > > > > >> > >> >> > > > > > > triggering a > > > > > > > > > > > > >> > >> >> > > > > > > > > > full > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > My first guess > is > > > that > > > > > the > > > > > > > > > options > > > > > > > > > > > > will > > > > > > > > > > > > >> be > > > > > > > > > > > > >> > >> >> easiest > > > > > > > > > > > > >> > >> >> > to > > > > > > > > > > > > >> > >> >> > > > do > > > > > > > > > > > > >> > >> >> > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > following > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > order: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.1 > > with > > > a > > > > > > > > dedicated > > > > > > > > > > > > >> > direct_memory > > > > > > > > > > > > >> > >> >> > > > parameter, > > > > > > > > > > > > >> > >> >> > > > > as > > > > > > > > > > > > >> > >> >> > > > > > > > > > discussed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > above. We would > > need > > > > to > > > > > > > find a > > > > > > > > > way > > > > > > > > > > > to > > > > > > > > > > > > >> set > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > > > > direct_memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > parameter > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > default. We > could > > > > start > > > > > > with > > > > > > > > 64 > > > > > > > > > MB > > > > > > > > > > > and > > > > > > > > > > > > >> see > > > > > > > > > > > > >> > >> how > > > > > > > > > > > > >> > >> >> it > > > > > > > > > > > > >> > >> >> > > goes > > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > practice. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > One > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > danger I see is > > that > > > > > > setting > > > > > > > > > this > > > > > > > > > > > loo > > > > > > > > > > > > >> low > > > > > > > > > > > > >> > can > > > > > > > > > > > > >> > >> >> > cause a > > > > > > > > > > > > >> > >> >> > > > > bunch > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > additional > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GCs compared to > > > before > > > > > (we > > > > > > > > need > > > > > > > > > to > > > > > > > > > > > > watch > > > > > > > > > > > > >> > this > > > > > > > > > > > > >> > >> >> > > > carefully). > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 2. It > > is > > > > > > actually > > > > > > > > > quite > > > > > > > > > > > > >> simple > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > >> >> > > implement, > > > > > > > > > > > > >> > >> >> > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > try > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > segfault safe we > > are > > > > at > > > > > > the > > > > > > > > > > moment. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > - Option 1.2: > We > > > > would > > > > > > not > > > > > > > > > touch > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > "-XX:MaxDirectMemorySize" > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > parameter > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > at all and > assume > > > that > > > > > all > > > > > > > the > > > > > > > > > > > direct > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > > allocations > > > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Netty do are > > > > infrequent > > > > > > > enough > > > > > > > > > to > > > > > > > > > > be > > > > > > > > > > > > >> > cleaned > > > > > > > > > > > > >> > >> up > > > > > > > > > > > > >> > >> >> > fast > > > > > > > > > > > > >> > >> >> > > > > enough > > > > > > > > > > > > >> > >> >> > > > > > > > > through > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > regular > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > GC. I am not > sure > > if > > > > > that > > > > > > > is a > > > > > > > > > > valid > > > > > > > > > > > > >> > >> assumption, > > > > > > > > > > > > >> > >> >> > > > though. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Best, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Stephan > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > 2:16 > > > > > > > > PM > > > > > > > > > > > > Xintong > > > > > > > > > > > > >> > Song > > > > > > > > > > > > >> > >> < > > > > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for > > sharing > > > > > your > > > > > > > > > opinion > > > > > > > > > > > > Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in > > favor > > > of > > > > > > > > > alternative > > > > > > > > > > > 2. > > > > > > > > > > > > I > > > > > > > > > > > > >> was > > > > > > > > > > > > >> > >> >> > wondering > > > > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > > > > Unsafe.allocate() > > > > > > > for > > > > > > > > > > > off-heap > > > > > > > > > > > > >> > >> managed > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. > > But > > > > > after > > > > > > > > > giving > > > > > > > > > > > it a > > > > > > > > > > > > >> > second > > > > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > > > > >> > >> >> > > > I > > > > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 > > > using > > > > > > direct > > > > > > > > > > memory > > > > > > > > > > > > for > > > > > > > > > > > > >> > >> off-heap > > > > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > > > > concern, > > > > > > I > > > > > > > > > think > > > > > > > > > > > what > > > > > > > > > > > > >> > >> proposed > > > > > > > > > > > > >> > >> >> in > > > > > > > > > > > > >> > >> >> > > this > > > > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap > managed > > > > > memory > > > > > > > and > > > > > > > > > > > network > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unsafe.allocate(), > > > > > which > > > > > > > > means > > > > > > > > > > > they > > > > > > > > > > > > >> are > > > > > > > > > > > > >> > >> >> > practically > > > > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM > > max > > > > > > direct > > > > > > > > > > memory. > > > > > > > > > > > > The > > > > > > > > > > > > >> > only > > > > > > > > > > > > >> > >> >> parts > > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct > > memory > > > > are > > > > > > task > > > > > > > > > > > off-heap > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> and > > > > > > > > > > > > >> > >> >> > JVM > > > > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly > > > alternative > > > > 2 > > > > > > > > suggests > > > > > > > > > > to > > > > > > > > > > > > set > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> JVM > > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug > 16, > > > 2019 > > > > > at > > > > > > > 1:48 > > > > > > > > > PM > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> Rohrmann > > > > > > > > > > > > >> > >> >> < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for > the > > > > > > > > clarification > > > > > > > > > > > > >> Xintong. I > > > > > > > > > > > > >> > >> >> > > understand > > > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be > in > > > > favour > > > > > > of > > > > > > > > > > option 2 > > > > > > > > > > > > >> > because > > > > > > > > > > > > >> > >> it > > > > > > > > > > > > >> > >> >> > makes > > > > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit > > the > > > > > direct > > > > > > > > > > memory, I > > > > > > > > > > > > >> fear > > > > > > > > > > > > >> > >> that > > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > > >> > >> >> > > might > > > > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as > > we > > > > are > > > > > > > > > currently > > > > > > > > > > > in: > > > > > > > > > > > > >> The > > > > > > > > > > > > >> > >> user > > > > > > > > > > > > >> > >> >> > might > > > > > > > > > > > > >> > >> >> > > > see > > > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by > the > > OS > > > > and > > > > > > > does > > > > > > > > > not > > > > > > > > > > > know > > > > > > > > > > > > >> why > > > > > > > > > > > > >> > >> this > > > > > > > > > > > > >> > >> >> is > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > > decrease > > > > > the > > > > > > > > > process > > > > > > > > > > > > memory > > > > > > > > > > > > >> > size > > > > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in > > order > > > to > > > > > > > > > accommodate > > > > > > > > > > > for > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> extra > > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > > decrease > > > > > > memory > > > > > > > > > > budgets > > > > > > > > > > > > >> which > > > > > > > > > > > > >> > >> are > > > > > > > > > > > > >> > >> >> not > > > > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the > > > overall > > > > > > > memory > > > > > > > > > > > > >> consumption. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug > > 16, > > > > 2019 > > > > > > at > > > > > > > > > 11:01 > > > > > > > > > > AM > > > > > > > > > > > > >> > Xintong > > > > > > > > > > > > >> > >> >> Song < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me > > explain > > > > > this > > > > > > > > with a > > > > > > > > > > > > >> concrete > > > > > > > > > > > > >> > >> >> example > > > > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say > we > > > > have > > > > > > the > > > > > > > > > > > following > > > > > > > > > > > > >> > >> scenario. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total > > Process > > > > > > Memory: > > > > > > > > 1GB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > > > > Memory > > > > > > > (Task > > > > > > > > > > > Off-Heap > > > > > > > > > > > > >> > >> Memory + > > > > > > > > > > > > >> > >> >> JVM > > > > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other > Memory > > > > (JVM > > > > > > Heap > > > > > > > > > > Memory, > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > Memory): > > > > > > 800MB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > > alternative > > > > 2, > > > > > > we > > > > > > > > set > > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > > alternative > > > > 3, > > > > > > we > > > > > > > > set > > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say > > 1TB. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > > actual > > > > > direct > > > > > > > > > memory > > > > > > > > > > > > usage > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > >> Task > > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not > > exceed > > > > > 200MB, > > > > > > > > then > > > > > > > > > > > > >> > alternative 2 > > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same > > utility. > > > > > > Setting > > > > > > > > > larger > > > > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of > the > > > > other > > > > > > > > memory > > > > > > > > > > > pools. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > > actual > > > > > direct > > > > > > > > > memory > > > > > > > > > > > > usage > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > >> Task > > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > > > > potentially > > > > > > > > > exceed > > > > > > > > > > > > 200MB, > > > > > > > > > > > > >> > then > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > > > > Alternative 2 > > > > > > > > suffers > > > > > > > > > > > from > > > > > > > > > > > > >> > >> frequent > > > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > > > >> > >> >> > > To > > > > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user > can > > do > > > > is > > > > > to > > > > > > > > > modify > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> configuration > > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > > > > Off-Heap > > > > > > > > Memory + > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> Overhead). > > > > > > > > > > > > >> > >> >> > Let's > > > > > > > > > > > > >> > >> >> > > > say > > > > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct > > > Memory > > > > > to > > > > > > > > 250MB, > > > > > > > > > > > this > > > > > > > > > > > > >> will > > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools > to > > > > 750MB, > > > > > > > given > > > > > > > > > the > > > > > > > > > > > > total > > > > > > > > > > > > >> > >> process > > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > > > > alternative > > > > > > > 3, > > > > > > > > > > there > > > > > > > > > > > is > > > > > > > > > > > > >> no > > > > > > > > > > > > >> > >> >> chance of > > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of > > > exceeding > > > > > the > > > > > > > > total > > > > > > > > > > > > process > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > limit, > > > > > > > > > > > > >> > >> >> > > > but > > > > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use > > up > > > > all > > > > > > the > > > > > > > > > > reserved > > > > > > > > > > > > >> native > > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, > > JVM > > > > > > > > Metaspace), > > > > > > > > > > if > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> actual > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet > very > > > > close > > > > > to > > > > > > > > > 200MB, > > > > > > > > > > > user > > > > > > > > > > > > >> > >> probably > > > > > > > > > > > > >> > >> >> do > > > > > > > > > > > > >> > >> >> > > not > > > > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Therefore, I > > > > think > > > > > > > from > > > > > > > > > the > > > > > > > > > > > > user's > > > > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for > > > alternative > > > > 2 > > > > > > may > > > > > > > > lead > > > > > > > > > > to > > > > > > > > > > > > >> lower > > > > > > > > > > > > >> > >> >> resource > > > > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > > 3. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, > Aug > > > 16, > > > > > 2019 > > > > > > > at > > > > > > > > > > 10:28 > > > > > > > > > > > AM > > > > > > > > > > > > >> Till > > > > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > > > > >> > >> >> > > < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess > > you > > > > have > > > > > > to > > > > > > > > help > > > > > > > > > > me > > > > > > > > > > > > >> > >> understand > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > > 2 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 > wrt > > to > > > > > > memory > > > > > > > > > under > > > > > > > > > > > > >> > utilization > > > > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > > Alternative > > > > 2: > > > > > > set > > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Overhead. > > > Then > > > > > > there > > > > > > > > is > > > > > > > > > > the > > > > > > > > > > > > risk > > > > > > > > > > > > >> > that > > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > > >> > >> >> > > size > > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of > > > garbage > > > > > > > > > collection > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> potentially > > > > > > > > > > > > >> > >> >> > an > > > > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > > Alternative > > > > 3: > > > > > > set > > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative > > > 2. > > > > > > This > > > > > > > > > would > > > > > > > > > > of > > > > > > > > > > > > >> course > > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How > would > > > > > > > alternative > > > > > > > > 2 > > > > > > > > > > now > > > > > > > > > > > > >> result > > > > > > > > > > > > >> > >> in an > > > > > > > > > > > > >> > >> >> > > under > > > > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared > > to > > > > > > > > alternative > > > > > > > > > 3? > > > > > > > > > > > If > > > > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > memory > > > > > size > > > > > > > and > > > > > > > > > we > > > > > > > > > > > use > > > > > > > > > > > > >> only > > > > > > > > > > > > >> > >> >> little, > > > > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative > > > 3 > > > > > > > results > > > > > > > > in > > > > > > > > > > > > memory > > > > > > > > > > > > >> > under > > > > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, > > Aug > > > > 13, > > > > > > 2019 > > > > > > > > at > > > > > > > > > > 4:19 > > > > > > > > > > > > PM > > > > > > > > > > > > >> > Yang > > > > > > > > > > > > >> > >> >> Wang < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > > > > xintong,till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Native > > > and > > > > > > > Direct > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My > point > > > is > > > > > > > setting > > > > > > > > a > > > > > > > > > > very > > > > > > > > > > > > >> large > > > > > > > > > > > > >> > >> max > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > differentiate > > > > > > > direct > > > > > > > > > and > > > > > > > > > > > > >> native > > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > > >> > >> >> > If > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > memory,including > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > > > > memory > > > > > > and > > > > > > > > > > > framework > > > > > > > > > > > > >> > direct > > > > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am > in > > > > favor > > > > > of > > > > > > > > > setting > > > > > > > > > > > > >> direct > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > with > > > > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > Calculation > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I > agree > > > with > > > > > > > > xintong. > > > > > > > > > > For > > > > > > > > > > > > Yarn > > > > > > > > > > > > >> > and > > > > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > > > > >> > >> >> > > > need > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > in > > > > > > > > > client > > > > > > > > > > > to > > > > > > > > > > > > >> avoid > > > > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the > > flink > > > > > > master. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong > > > > Song < > > > > > > > > > > > > >> > >> [hidden email] > > > > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > > > > replying, > > > > > > > > > > > Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > About > > > > > > > > > MemorySegment, I > > > > > > > > > > > > think > > > > > > > > > > > > >> > you > > > > > > > > > > > > >> > >> are > > > > > > > > > > > > >> > >> >> > > right > > > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > issue > > in > > > > the > > > > > > > scope > > > > > > > > > of > > > > > > > > > > > this > > > > > > > > > > > > >> > FLIP. > > > > > > > > > > > > >> > >> >> This > > > > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > configure > > > > > > memory > > > > > > > > > pools > > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > > > > consumers > > > > > > > > use > > > > > > > > > > it. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > About > > > > direct > > > > > > > > > memory, I > > > > > > > > > > > > think > > > > > > > > > > > > >> > >> >> > alternative > > > > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > reservation > > > > > > > issue > > > > > > > > > that > > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > > >> > >> >> > > does, > > > > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > using > > > > memory > > > > > > at > > > > > > > > the > > > > > > > > > > > > >> container > > > > > > > > > > > > >> > >> level, > > > > > > > > > > > > >> > >> >> > > which > > > > > > > > > > > > >> > >> >> > > > is > > > > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both > > > "Task > > > > > > > > Off-Heap > > > > > > > > > > > > Memory" > > > > > > > > > > > > >> and > > > > > > > > > > > > >> > >> "JVM > > > > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > > 2, > > > > > > > > users > > > > > > > > > > > might > > > > > > > > > > > > >> > >> configure > > > > > > > > > > > > >> > >> >> > them > > > > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just > > to > > > > > avoid > > > > > > > > > getting > > > > > > > > > > a > > > > > > > > > > > > >> direct > > > > > > > > > > > > >> > >> OOM. > > > > > > > > > > > > >> > >> >> For > > > > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > > OOM, > > > > > so > > > > > > > > they > > > > > > > > > > may > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> config > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > two > > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > > > > consequences > > > > > > > > are > > > > > > > > > > > risks > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > >> >> overall > > > > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thank > > > you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On > > Tue, > > > > Aug > > > > > > 13, > > > > > > > > 2019 > > > > > > > > > > at > > > > > > > > > > > > >> 9:39 AM > > > > > > > > > > > > >> > >> Till > > > > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > for > > > > > > > > > proposing > > > > > > > > > > > > this > > > > > > > > > > > > >> > FLIP > > > > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > All > > in > > > > > all I > > > > > > > > think > > > > > > > > > > it > > > > > > > > > > > > >> already > > > > > > > > > > > > >> > >> >> looks > > > > > > > > > > > > >> > >> >> > > quite > > > > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > question > > > > > > about > > > > > > > > > > > > allocating > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > strictly > > > > > > > > necessary > > > > > > > > > > to > > > > > > > > > > > do > > > > > > > > > > > > >> in > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > context > > > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > > > done > > > > > as a > > > > > > > > > follow > > > > > > > > > > > up? > > > > > > > > > > > > >> > Without > > > > > > > > > > > > >> > >> >> > knowing > > > > > > > > > > > > >> > >> >> > > > all > > > > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > that > > > we > > > > > > would > > > > > > > > > widen > > > > > > > > > > > the > > > > > > > > > > > > >> scope > > > > > > > > > > > > >> > >> of > > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to > > > touch > > > > > all > > > > > > > the > > > > > > > > > > > > existing > > > > > > > > > > > > >> > call > > > > > > > > > > > > >> > >> >> sites > > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > segments > > > > > > > > > > (this > > > > > > > > > > > > >> should > > > > > > > > > > > > >> > >> >> mainly > > > > > > > > > > > > >> > >> >> > be > > > > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > the > > > > memory > > > > > > > > > > reservation > > > > > > > > > > > > >> call > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > this > > > > and I > > > > > > > would > > > > > > > > > > hope > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> this is > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > streaming > > > > > > job > > > > > > > > > would > > > > > > > > > > > have > > > > > > > > > > > > >> with > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Concerning > > > > > > the > > > > > > > > > > second > > > > > > > > > > > > open > > > > > > > > > > > > >> > >> >> question > > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > memory > > > > > > > > > > limit, I > > > > > > > > > > > > >> would > > > > > > > > > > > > >> > >> also > > > > > > > > > > > > >> > >> >> be > > > > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > leaving > > > > it > > > > > > > open > > > > > > > > > > would > > > > > > > > > > > be > > > > > > > > > > > > >> > best. > > > > > > > > > > > > >> > >> My > > > > > > > > > > > > >> > >> >> > > concern > > > > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > > in > > > a > > > > > > > similar > > > > > > > > > > > > situation > > > > > > > > > > > > >> as > > > > > > > > > > > > >> > we > > > > > > > > > > > > >> > >> >> are > > > > > > > > > > > > >> > >> >> > now > > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > the > > > > > > different > > > > > > > > > memory > > > > > > > > > > > > pools > > > > > > > > > > > > >> > are > > > > > > > > > > > > >> > >> not > > > > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > > > > different > > > > > > > > pool, > > > > > > > > > > then > > > > > > > > > > > > it > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > >> quite > > > > > > > > > > > > >> > >> >> > hard > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > process > > > > to > > > > > > get > > > > > > > > > > killed > > > > > > > > > > > > for > > > > > > > > > > > > >> > using > > > > > > > > > > > > >> > >> >> too > > > > > > > > > > > > >> > >> >> > > much > > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > lead > > > to > > > > a > > > > > > > > similar > > > > > > > > > > > > >> situation > > > > > > > > > > > > >> > >> what > > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > > >> > >> >> > > have > > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > setting > > > > a > > > > > > sane > > > > > > > > > > default > > > > > > > > > > > > >> value > > > > > > > > > > > > >> > >> for > > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > option > > > > to > > > > > > > > increase > > > > > > > > > > it > > > > > > > > > > > if > > > > > > > > > > > > >> he > > > > > > > > > > > > >> > >> runs > > > > > > > > > > > > >> > >> >> into > > > > > > > > > > > > >> > >> >> > > an > > > > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > > > > how > > > > > > > > > would > > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > > >> > >> >> > lead > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > > > 3 > > > > > > > > > where > > > > > > > > > > we > > > > > > > > > > > > set > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> direct > > > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On > > > Fri, > > > > > Aug > > > > > > 9, > > > > > > > > > 2019 > > > > > > > > > > at > > > > > > > > > > > > >> 9:12 > > > > > > > > > > > > >> > AM > > > > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > for > > > > > > > the > > > > > > > > > > > > feedback, > > > > > > > > > > > > >> > >> Yang. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > > > > your > > > > > > > > > > > > comments: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > *Native > > > > > > and > > > > > > > > > Direct > > > > > > > > > > > > >> Memory* > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > I > > > > think > > > > > > > > setting > > > > > > > > > a > > > > > > > > > > > very > > > > > > > > > > > > >> > large > > > > > > > > > > > > >> > >> max > > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > good > > > > > > sides. > > > > > > > > > E.g., > > > > > > > > > > we > > > > > > > > > > > > do > > > > > > > > > > > > >> not > > > > > > > > > > > > >> > >> >> worry > > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > to > > > > > > allocate > > > > > > > > > > managed > > > > > > > > > > > / > > > > > > > > > > > > >> > network > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate() > > . > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > However, > > > > > > > there > > > > > > > > > are > > > > > > > > > > > > also > > > > > > > > > > > > >> > some > > > > > > > > > > > > >> > >> >> down > > > > > > > > > > > > >> > >> >> > > sides > > > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > One > > > > > > > > thing I > > > > > > > > > > can > > > > > > > > > > > > >> think > > > > > > > > > > > > >> > >> of is > > > > > > > > > > > > >> > >> >> > that > > > > > > > > > > > > >> > >> >> > > > if > > > > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > killed > > > > > > > due > > > > > > > > to > > > > > > > > > > > > >> overusing > > > > > > > > > > > > >> > >> >> memory, > > > > > > > > > > > > >> > >> >> > it > > > > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > the > > > > > > > > memory > > > > > > > > > > is > > > > > > > > > > > > >> > overused. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Another > > > > > > > > > down > > > > > > > > > > > side > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > >> that > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > > > memory > > > > > > > > > > > > limit, > > > > > > > > > > > > >> > >> because > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > means > > > > > > we > > > > > > > > kind > > > > > > > > > > of > > > > > > > > > > > > >> relay > > > > > > > > > > > > >> > on > > > > > > > > > > > > >> > >> >> heap > > > > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > > > > That > > > > > > > > > > > could > > > > > > > > > > > > >> be a > > > > > > > > > > > > >> > >> >> problem > > > > > > > > > > > > >> > >> >> > in > > > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > usage > > > > > > but > > > > > > > > not > > > > > > > > > > > > enough > > > > > > > > > > > > >> > heap > > > > > > > > > > > > >> > >> >> > activity > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Maybe > > > > > you > > > > > > > can > > > > > > > > > > share > > > > > > > > > > > > your > > > > > > > > > > > > >> > >> reasons > > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > if > > > > there > > > > > > are > > > > > > > > > > > anything > > > > > > > > > > > > >> else > > > > > > > > > > > > >> > I > > > > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > > > > Calculation* > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If > > > > there > > > > > > is > > > > > > > > any > > > > > > > > > > > > conflict > > > > > > > > > > > > >> > >> between > > > > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > > > > specified, > > > > > > > > > > > > I > > > > > > > > > > > > >> > >> think we > > > > > > > > > > > > >> > >> >> > > should > > > > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > I > > > > think > > > > > > > doing > > > > > > > > > > > checking > > > > > > > > > > > > >> on > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > client > > > > > > > > > > > > >> > >> >> > > > side > > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > K8s > > > we > > > > > can > > > > > > > > > > discover > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> problem > > > > > > > > > > > > >> > >> >> > > before > > > > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > > > > always > > > > > > a > > > > > > > > good > > > > > > > > > > > > thing. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > But > > > we > > > > > can > > > > > > > not > > > > > > > > > > only > > > > > > > > > > > > >> rely on > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > client > > > > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > > > > cluster > > > > > > > > > > > > >> > >> TaskManagers > > > > > > > > > > > > >> > >> >> on > > > > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > > > > and > > > > > > > > > > > the > > > > > > > > > > > > >> > client > > > > > > > > > > > > >> > >> >> does > > > > > > > > > > > > >> > >> >> > > see > > > > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > What > > > > do > > > > > > you > > > > > > > > > think? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > On > > > > Thu, > > > > > > Aug > > > > > > > 8, > > > > > > > > > > 2019 > > > > > > > > > > > at > > > > > > > > > > > > >> 5:09 > > > > > > > > > > > > >> > >> PM > > > > > > > > > > > > >> > >> >> Yang > > > > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > xintong, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > for > > > > > > > > > your > > > > > > > > > > > > >> detailed > > > > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > > > > it > > > > > > > > > > > will > > > > > > > > > > > > be > > > > > > > > > > > > >> > more > > > > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > just > > > > > > have > > > > > > > > few > > > > > > > > > > > > >> questions > > > > > > > > > > > > >> > >> about > > > > > > > > > > > > >> > >> >> it. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Native > > > > > > > > > and > > > > > > > > > > > > Direct > > > > > > > > > > > > >> > >> Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > do > > > > > > not > > > > > > > > > > > > >> differentiate > > > > > > > > > > > > >> > >> user > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > included > > > > > > > in > > > > > > > > > task > > > > > > > > > > > > >> off-heap > > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Memory > > > > > > > > > > > > >> Calculation > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > If > > > > the > > > > > > sum > > > > > > > > of > > > > > > > > > > and > > > > > > > > > > > > >> > >> fine-grained > > > > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > larger > > > > > > > > than > > > > > > > > > > > total > > > > > > > > > > > > >> > >> process > > > > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > need > > > > > > to > > > > > > > > > check > > > > > > > > > > > the > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > Song < > > > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > > everyone, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > > > would > > > > > > > > > like > > > > > > > > > > to > > > > > > > > > > > > >> start > > > > > > > > > > > > >> > a > > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > > > > design > > > > > > > > > > > > "Memory > > > > > > > > > > > > >> > >> >> Management > > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > updates > > > > > > > > > > > from > > > > > > > > > > > > >> > >> follow-up > > > > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This > > > > > > > FLIP > > > > > > > > > > > > addresses > > > > > > > > > > > > >> > >> several > > > > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > Different > > > > > > > > > > > > >> > >> configuration > > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Complex > > > > > > > > > > and > > > > > > > > > > > > >> > >> difficult > > > > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > Complicated, > > > > > > > > > > > > >> > >> uncertain > > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > > > > changes > > > > > > > > > to > > > > > > > > > > > > solve > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > problems > > > > > > > > > > > > >> > >> >> > > > can > > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Extend > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> manager > > > > > > > > > > > > >> > >> >> to > > > > > > > > > > > > >> > >> >> > > also > > > > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Modify > > > > > > > > > > how > > > > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> reservations > > > > > > > > > > > > >> > >> and > > > > > > > > > > > > >> > >> >> > pools. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > Simplify > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > > > > find > > > > > > > > > > more > > > > > > > > > > > > >> > details > > > > > > > > > > > > >> > >> in > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > > > > note > > > > > > > > > > > that > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> early > > > > > > > > > > > > >> > >> >> > > design > > > > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > > > > to > > > > > > > > > > > > have > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > > > > forward > > > > > > > > > > > to > > > > > > > > > > > > >> your > > > > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > > > you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > On Fri, Aug 16, > > 2019 > > > > at > > > > > > 2:16 > > > > > > > > PM > > > > > > > > > > > > Xintong > > > > > > > > > > > > >> > Song > > > > > > > > > > > > >> > >> < > > > > > > > > > > > > >> > >> >> > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thanks for > > sharing > > > > > your > > > > > > > > > opinion > > > > > > > > > > > > Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > I'm also in > > favor > > > of > > > > > > > > > alternative > > > > > > > > > > > 2. > > > > > > > > > > > > I > > > > > > > > > > > > >> was > > > > > > > > > > > > >> > >> >> > wondering > > > > > > > > > > > > >> > >> >> > > > > > whether > > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > can > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > avoid > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > using > > > > > Unsafe.allocate() > > > > > > > for > > > > > > > > > > > off-heap > > > > > > > > > > > > >> > >> managed > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > network > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3. > > But > > > > > after > > > > > > > > > giving > > > > > > > > > > > it a > > > > > > > > > > > > >> > second > > > > > > > > > > > > >> > >> >> > > thought, > > > > > > > > > > > > >> > >> >> > > > I > > > > > > > > > > > > >> > >> >> > > > > > > think > > > > > > > > > > > > >> > >> >> > > > > > > > > even > > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > alternative 3 > > > using > > > > > > direct > > > > > > > > > > memory > > > > > > > > > > > > for > > > > > > > > > > > > >> > >> off-heap > > > > > > > > > > > > >> > >> >> > > > managed > > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > cause > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > problems. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Regarding your > > > > > concern, > > > > > > I > > > > > > > > > think > > > > > > > > > > > what > > > > > > > > > > > > >> > >> proposed > > > > > > > > > > > > >> > >> >> in > > > > > > > > > > > > >> > >> >> > > this > > > > > > > > > > > > >> > >> >> > > > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > both > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > off-heap > managed > > > > > memory > > > > > > > and > > > > > > > > > > > network > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > allocated > > > > > > > > > > > > >> > >> >> > > > > > > through > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unsafe.allocate(), > > > > > which > > > > > > > > means > > > > > > > > > > > they > > > > > > > > > > > > >> are > > > > > > > > > > > > >> > >> >> > practically > > > > > > > > > > > > >> > >> >> > > > > > native > > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > limited by JVM > > max > > > > > > direct > > > > > > > > > > memory. > > > > > > > > > > > > The > > > > > > > > > > > > >> > only > > > > > > > > > > > > >> > >> >> parts > > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > limited > > > > > > > > > > > > >> > >> >> > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max direct > > memory > > > > are > > > > > > task > > > > > > > > > > > off-heap > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> and > > > > > > > > > > > > >> > >> >> > JVM > > > > > > > > > > > > >> > >> >> > > > > > > overhead, > > > > > > > > > > > > >> > >> >> > > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > exactly > > > alternative > > > > 2 > > > > > > > > suggests > > > > > > > > > > to > > > > > > > > > > > > set > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> JVM > > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > to. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > On Fri, Aug > 16, > > > 2019 > > > > > at > > > > > > > 1:48 > > > > > > > > > PM > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> Rohrmann > > > > > > > > > > > > >> > >> >> < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Thanks for > the > > > > > > > > clarification > > > > > > > > > > > > >> Xintong. I > > > > > > > > > > > > >> > >> >> > > understand > > > > > > > > > > > > >> > >> >> > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > two > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > alternatives > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > now. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > I would be > in > > > > favour > > > > > > of > > > > > > > > > > option 2 > > > > > > > > > > > > >> > because > > > > > > > > > > > > >> > >> it > > > > > > > > > > > > >> > >> >> > makes > > > > > > > > > > > > >> > >> >> > > > > > things > > > > > > > > > > > > >> > >> >> > > > > > > > > > > explicit. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > If > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't limit > > the > > > > > direct > > > > > > > > > > memory, I > > > > > > > > > > > > >> fear > > > > > > > > > > > > >> > >> that > > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > > >> > >> >> > > might > > > > > > > > > > > > >> > >> >> > > > > end > > > > > > > > > > > > >> > >> >> > > > > > > up > > > > > > > > > > > > >> > >> >> > > > > > > > > in a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > similar > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > situation as > > we > > > > are > > > > > > > > > currently > > > > > > > > > > > in: > > > > > > > > > > > > >> The > > > > > > > > > > > > >> > >> user > > > > > > > > > > > > >> > >> >> > might > > > > > > > > > > > > >> > >> >> > > > see > > > > > > > > > > > > >> > >> >> > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > her > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > process > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > gets > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > killed by > the > > OS > > > > and > > > > > > > does > > > > > > > > > not > > > > > > > > > > > know > > > > > > > > > > > > >> why > > > > > > > > > > > > >> > >> this > > > > > > > > > > > > >> > >> >> is > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > case. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Consequently, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > > decrease > > > > > the > > > > > > > > > process > > > > > > > > > > > > memory > > > > > > > > > > > > >> > size > > > > > > > > > > > > >> > >> >> > > (similar > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > increasing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > cutoff > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > ratio) in > > order > > > to > > > > > > > > > accommodate > > > > > > > > > > > for > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> extra > > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > > Even > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > worse, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > she > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > tries to > > > decrease > > > > > > memory > > > > > > > > > > budgets > > > > > > > > > > > > >> which > > > > > > > > > > > > >> > >> are > > > > > > > > > > > > >> > >> >> not > > > > > > > > > > > > >> > >> >> > > > fully > > > > > > > > > > > > >> > >> >> > > > > > used > > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > hence > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > won't > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > change the > > > overall > > > > > > > memory > > > > > > > > > > > > >> consumption. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > On Fri, Aug > > 16, > > > > 2019 > > > > > > at > > > > > > > > > 11:01 > > > > > > > > > > AM > > > > > > > > > > > > >> > Xintong > > > > > > > > > > > > >> > >> >> Song < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let me > > explain > > > > > this > > > > > > > > with a > > > > > > > > > > > > >> concrete > > > > > > > > > > > > >> > >> >> example > > > > > > > > > > > > >> > >> >> > > Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Let's say > we > > > > have > > > > > > the > > > > > > > > > > > following > > > > > > > > > > > > >> > >> scenario. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Total > > Process > > > > > > Memory: > > > > > > > > 1GB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM Direct > > > > Memory > > > > > > > (Task > > > > > > > > > > > Off-Heap > > > > > > > > > > > > >> > >> Memory + > > > > > > > > > > > > >> > >> >> JVM > > > > > > > > > > > > >> > >> >> > > > > > > Overhead): > > > > > > > > > > > > >> > >> >> > > > > > > > > > 200MB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Other > Memory > > > > (JVM > > > > > > Heap > > > > > > > > > > Memory, > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> Metaspace, > > > > > > > > > > > > >> > >> >> > > > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > Memory): > > > > > > 800MB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > > alternative > > > > 2, > > > > > > we > > > > > > > > set > > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > 200MB. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > For > > > alternative > > > > 3, > > > > > > we > > > > > > > > set > > > > > > > > > > > > >> > >> >> > > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > very > > > > > > > > > > > > >> > >> >> > > > > > > > > > > large > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > value, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > let's say > > 1TB. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > > actual > > > > > direct > > > > > > > > > memory > > > > > > > > > > > > usage > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > >> Task > > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > Overhead > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > do not > > exceed > > > > > 200MB, > > > > > > > > then > > > > > > > > > > > > >> > alternative 2 > > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > > >> > >> >> > > > > > > alternative 3 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > same > > utility. > > > > > > Setting > > > > > > > > > larger > > > > > > > > > > > > >> > >> >> > > > > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > > > > will > > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > reduce > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > sizes of > the > > > > other > > > > > > > > memory > > > > > > > > > > > pools. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > If the > > actual > > > > > direct > > > > > > > > > memory > > > > > > > > > > > > usage > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > >> Task > > > > > > > > > > > > >> > >> >> > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Overhead > > > > > potentially > > > > > > > > > exceed > > > > > > > > > > > > 200MB, > > > > > > > > > > > > >> > then > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - > > > > Alternative 2 > > > > > > > > suffers > > > > > > > > > > > from > > > > > > > > > > > > >> > >> frequent > > > > > > > > > > > > >> > >> >> OOM. > > > > > > > > > > > > >> > >> >> > > To > > > > > > > > > > > > >> > >> >> > > > > > avoid > > > > > > > > > > > > >> > >> >> > > > > > > > > that, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > only > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > thing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > user > can > > do > > > > is > > > > > to > > > > > > > > > modify > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> configuration > > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > > >> > >> >> > > > > > > > increase > > > > > > > > > > > > >> > >> >> > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > (Task > > > > Off-Heap > > > > > > > > Memory + > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> Overhead). > > > > > > > > > > > > >> > >> >> > Let's > > > > > > > > > > > > >> > >> >> > > > say > > > > > > > > > > > > >> > >> >> > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > increases > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Direct > > > Memory > > > > > to > > > > > > > > 250MB, > > > > > > > > > > > this > > > > > > > > > > > > >> will > > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > > total > > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > pools > to > > > > 750MB, > > > > > > > given > > > > > > > > > the > > > > > > > > > > > > total > > > > > > > > > > > > >> > >> process > > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > > >> > >> >> > > > > > > remains > > > > > > > > > > > > >> > >> >> > > > > > > > > > 1GB. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > - For > > > > > alternative > > > > > > > 3, > > > > > > > > > > there > > > > > > > > > > > is > > > > > > > > > > > > >> no > > > > > > > > > > > > >> > >> >> chance of > > > > > > > > > > > > >> > >> >> > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > OOM. > > > > > > > > > > > > >> > >> >> > > > > > > > > > There > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > chances > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > of > > > exceeding > > > > > the > > > > > > > > total > > > > > > > > > > > > process > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > limit, > > > > > > > > > > > > >> > >> >> > > > but > > > > > > > > > > > > >> > >> >> > > > > > > given > > > > > > > > > > > > >> > >> >> > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > process > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > not use > > up > > > > all > > > > > > the > > > > > > > > > > reserved > > > > > > > > > > > > >> native > > > > > > > > > > > > >> > >> >> memory > > > > > > > > > > > > >> > >> >> > > > > > (Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > > > > Managed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > Memory, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Network > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Memory, > > JVM > > > > > > > > Metaspace), > > > > > > > > > > if > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> actual > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > usage > > > > > > > > > > > > >> > >> >> > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > slightly > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > above > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > yet > very > > > > close > > > > > to > > > > > > > > > 200MB, > > > > > > > > > > > user > > > > > > > > > > > > >> > >> probably > > > > > > > > > > > > >> > >> >> do > > > > > > > > > > > > >> > >> >> > > not > > > > > > > > > > > > >> > >> >> > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > change > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > configurations. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Therefore, I > > > > think > > > > > > > from > > > > > > > > > the > > > > > > > > > > > > user's > > > > > > > > > > > > >> > >> >> > > perspective, a > > > > > > > > > > > > >> > >> >> > > > > > > > feasible > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > for > > > alternative > > > > 2 > > > > > > may > > > > > > > > lead > > > > > > > > > > to > > > > > > > > > > > > >> lower > > > > > > > > > > > > >> > >> >> resource > > > > > > > > > > > > >> > >> >> > > > > > > utilization > > > > > > > > > > > > >> > >> >> > > > > > > > > > > compared > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > > 3. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Xintong > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > On Fri, > Aug > > > 16, > > > > > 2019 > > > > > > > at > > > > > > > > > > 10:28 > > > > > > > > > > > AM > > > > > > > > > > > > >> Till > > > > > > > > > > > > >> > >> >> > Rohrmann > > > > > > > > > > > > >> > >> >> > > < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > I guess > > you > > > > have > > > > > > to > > > > > > > > help > > > > > > > > > > me > > > > > > > > > > > > >> > >> understand > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > > > difference > > > > > > > > > > > > >> > >> >> > > > > > > > > > between > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > alternative > > 2 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > and 3 > wrt > > to > > > > > > memory > > > > > > > > > under > > > > > > > > > > > > >> > utilization > > > > > > > > > > > > >> > >> >> > > Xintong. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > > Alternative > > > > 2: > > > > > > set > > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > Task > > > > > > > > > > > > >> > >> >> > > > > > > > > > Off-Heap > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > JVM > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Overhead. > > > Then > > > > > > there > > > > > > > > is > > > > > > > > > > the > > > > > > > > > > > > risk > > > > > > > > > > > > >> > that > > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > > >> > >> >> > > size > > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > too > > > > > > > > > > > > >> > >> >> > > > > > > > > low > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > resulting > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > in a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > lot of > > > garbage > > > > > > > > > collection > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> potentially > > > > > > > > > > > > >> > >> >> > an > > > > > > > > > > > > >> > >> >> > > > OOM. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > - > > > Alternative > > > > 3: > > > > > > set > > > > > > > > > > > > >> > >> >> XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > > > > something > > > > > > > > > > > > >> > >> >> > > > > > > > > > > larger > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative > > > 2. > > > > > > This > > > > > > > > > would > > > > > > > > > > of > > > > > > > > > > > > >> course > > > > > > > > > > > > >> > >> >> reduce > > > > > > > > > > > > >> > >> >> > > the > > > > > > > > > > > > >> > >> >> > > > > > sizes > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > other > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > types. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > How > would > > > > > > > alternative > > > > > > > > 2 > > > > > > > > > > now > > > > > > > > > > > > >> result > > > > > > > > > > > > >> > >> in an > > > > > > > > > > > > >> > >> >> > > under > > > > > > > > > > > > >> > >> >> > > > > > > > > utilization > > > > > > > > > > > > >> > >> >> > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > compared > > to > > > > > > > > alternative > > > > > > > > > 3? > > > > > > > > > > > If > > > > > > > > > > > > >> > >> >> alternative 3 > > > > > > > > > > > > >> > >> >> > > > > > strictly > > > > > > > > > > > > >> > >> >> > > > > > > > > sets a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > higher > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > max > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > memory > > > > > size > > > > > > > and > > > > > > > > > we > > > > > > > > > > > use > > > > > > > > > > > > >> only > > > > > > > > > > > > >> > >> >> little, > > > > > > > > > > > > >> > >> >> > > > then I > > > > > > > > > > > > >> > >> >> > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > expect > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > alternative > > > 3 > > > > > > > results > > > > > > > > in > > > > > > > > > > > > memory > > > > > > > > > > > > >> > under > > > > > > > > > > > > >> > >> >> > > > > utilization. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > On Tue, > > Aug > > > > 13, > > > > > > 2019 > > > > > > > > at > > > > > > > > > > 4:19 > > > > > > > > > > > > PM > > > > > > > > > > > > >> > Yang > > > > > > > > > > > > >> > >> >> Wang < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Hi > > > > > xintong,till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Native > > > and > > > > > > > Direct > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > My > point > > > is > > > > > > > setting > > > > > > > > a > > > > > > > > > > very > > > > > > > > > > > > >> large > > > > > > > > > > > > >> > >> max > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > size > > > > > > > > > > > > >> > >> >> > > > > > > > > > > when > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > differentiate > > > > > > > direct > > > > > > > > > and > > > > > > > > > > > > >> native > > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > > >> > >> >> > If > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > memory,including > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > direct > > > > memory > > > > > > and > > > > > > > > > > > framework > > > > > > > > > > > > >> > direct > > > > > > > > > > > > >> > >> >> > > > memory,could > > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > calculated > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > correctly,then > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > i am > in > > > > favor > > > > > of > > > > > > > > > setting > > > > > > > > > > > > >> direct > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > with > > > > > > > > > > > > >> > >> >> > > > > fixed > > > > > > > > > > > > >> > >> >> > > > > > > > > value. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > Calculation > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > I > agree > > > with > > > > > > > > xintong. > > > > > > > > > > For > > > > > > > > > > > > Yarn > > > > > > > > > > > > >> > and > > > > > > > > > > > > >> > >> >> k8s,we > > > > > > > > > > > > >> > >> >> > > > need > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > check > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > in > > > > > > > > > client > > > > > > > > > > > to > > > > > > > > > > > > >> avoid > > > > > > > > > > > > >> > >> >> > > submitting > > > > > > > > > > > > >> > >> >> > > > > > > > > successfully > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > failing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > the > > flink > > > > > > master. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Yang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > Xintong > > > > Song < > > > > > > > > > > > > >> > >> [hidden email] > > > > > > > > > > > > >> > >> >> > > > > >于2019年8月13日 > > > > > > > > > > > > >> > >> >> > > > > > > > > > 周二22:07写道： > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thanks > > > for > > > > > > > > replying, > > > > > > > > > > > Till. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > About > > > > > > > > > MemorySegment, I > > > > > > > > > > > > think > > > > > > > > > > > > >> > you > > > > > > > > > > > > >> > >> are > > > > > > > > > > > > >> > >> >> > > right > > > > > > > > > > > > >> > >> >> > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > include > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > issue > > in > > > > the > > > > > > > scope > > > > > > > > > of > > > > > > > > > > > this > > > > > > > > > > > > >> > FLIP. > > > > > > > > > > > > >> > >> >> This > > > > > > > > > > > > >> > >> >> > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > concentrate > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > configure > > > > > > memory > > > > > > > > > pools > > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > TaskExecutors, > > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > minimum > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > involvement > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > memory > > > > > > consumers > > > > > > > > use > > > > > > > > > > it. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > About > > > > direct > > > > > > > > > memory, I > > > > > > > > > > > > think > > > > > > > > > > > > >> > >> >> > alternative > > > > > > > > > > > > >> > >> >> > > 3 > > > > > > > > > > > > >> > >> >> > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > having > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > same > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > over > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > reservation > > > > > > > issue > > > > > > > > > that > > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > > >> > >> >> > > does, > > > > > > > > > > > > >> > >> >> > > > > but > > > > > > > > > > > > >> > >> >> > > > > > at > > > > > > > > > > > > >> > >> >> > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > cost > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > risk > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > using > > > > memory > > > > > > at > > > > > > > > the > > > > > > > > > > > > >> container > > > > > > > > > > > > >> > >> level, > > > > > > > > > > > > >> > >> >> > > which > > > > > > > > > > > > >> > >> >> > > > is > > > > > > > > > > > > >> > >> >> > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > good. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > My > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > point > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > both > > > "Task > > > > > > > > Off-Heap > > > > > > > > > > > > Memory" > > > > > > > > > > > > >> and > > > > > > > > > > > > >> > >> "JVM > > > > > > > > > > > > >> > >> >> > > > > Overhead" > > > > > > > > > > > > >> > >> >> > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > easy > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > config. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > For > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > > 2, > > > > > > > > users > > > > > > > > > > > might > > > > > > > > > > > > >> > >> configure > > > > > > > > > > > > >> > >> >> > them > > > > > > > > > > > > >> > >> >> > > > > > higher > > > > > > > > > > > > >> > >> >> > > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > > > what > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > actually > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > needed, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > just > > to > > > > > avoid > > > > > > > > > getting > > > > > > > > > > a > > > > > > > > > > > > >> direct > > > > > > > > > > > > >> > >> OOM. > > > > > > > > > > > > >> > >> >> For > > > > > > > > > > > > >> > >> >> > > > > > > alternative > > > > > > > > > > > > >> > >> >> > > > > > > > > 3, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > users > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > get > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > > OOM, > > > > > so > > > > > > > > they > > > > > > > > > > may > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> config > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > two > > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > aggressively > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > high. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > But > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > the > > > > > > consequences > > > > > > > > are > > > > > > > > > > > risks > > > > > > > > > > > > >> of > > > > > > > > > > > > >> > >> >> overall > > > > > > > > > > > > >> > >> >> > > > > container > > > > > > > > > > > > >> > >> >> > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > usage > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exceeds > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > budget. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > Thank > > > you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Xintong > > > > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > On > > Tue, > > > > Aug > > > > > > 13, > > > > > > > > 2019 > > > > > > > > > > at > > > > > > > > > > > > >> 9:39 AM > > > > > > > > > > > > >> > >> Till > > > > > > > > > > > > >> > >> >> > > > > Rohrmann < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > for > > > > > > > > > proposing > > > > > > > > > > > > this > > > > > > > > > > > > >> > FLIP > > > > > > > > > > > > >> > >> >> > Xintong. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > All > > in > > > > > all I > > > > > > > > think > > > > > > > > > > it > > > > > > > > > > > > >> already > > > > > > > > > > > > >> > >> >> looks > > > > > > > > > > > > >> > >> >> > > quite > > > > > > > > > > > > >> > >> >> > > > > > good. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Concerning > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > first > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > open > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > question > > > > > > about > > > > > > > > > > > > allocating > > > > > > > > > > > > >> > >> memory > > > > > > > > > > > > >> > >> >> > > > segments, > > > > > > > > > > > > >> > >> >> > > > > I > > > > > > > > > > > > >> > >> >> > > > > > > was > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > wondering > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > whether > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > strictly > > > > > > > > necessary > > > > > > > > > > to > > > > > > > > > > > do > > > > > > > > > > > > >> in > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > context > > > > > > > > > > > > >> > >> >> > > > of > > > > > > > > > > > > >> > >> >> > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > > > > > or > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > whether > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > > > done > > > > > as a > > > > > > > > > follow > > > > > > > > > > > up? > > > > > > > > > > > > >> > Without > > > > > > > > > > > > >> > >> >> > knowing > > > > > > > > > > > > >> > >> >> > > > all > > > > > > > > > > > > >> > >> >> > > > > > > > > details, > > > > > > > > > > > > >> > >> >> > > > > > > > > > I > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > concerned > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > that > > > we > > > > > > would > > > > > > > > > widen > > > > > > > > > > > the > > > > > > > > > > > > >> scope > > > > > > > > > > > > >> > >> of > > > > > > > > > > > > >> > >> >> this > > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > too > > > > > > > > > > > > >> > >> >> > > > > > > > much > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > because > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > to > > > touch > > > > > all > > > > > > > the > > > > > > > > > > > > existing > > > > > > > > > > > > >> > call > > > > > > > > > > > > >> > >> >> sites > > > > > > > > > > > > >> > >> >> > of > > > > > > > > > > > > >> > >> >> > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > MemoryManager > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > where > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > allocate > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > segments > > > > > > > > > > (this > > > > > > > > > > > > >> should > > > > > > > > > > > > >> > >> >> mainly > > > > > > > > > > > > >> > >> >> > be > > > > > > > > > > > > >> > >> >> > > > > batch > > > > > > > > > > > > >> > >> >> > > > > > > > > > > operators). > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > The > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > addition > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > the > > > > memory > > > > > > > > > > reservation > > > > > > > > > > > > >> call > > > > > > > > > > > > >> > to > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager > > > > > > > > > > > > >> > >> >> > > > > > > > > > should > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > affected > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > this > > > > and I > > > > > > > would > > > > > > > > > > hope > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> this is > > > > > > > > > > > > >> > >> >> > the > > > > > > > > > > > > >> > >> >> > > > only > > > > > > > > > > > > >> > >> >> > > > > > > point > > > > > > > > > > > > >> > >> >> > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > interaction > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > streaming > > > > > > job > > > > > > > > > would > > > > > > > > > > > have > > > > > > > > > > > > >> with > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > > > > > MemoryManager. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Concerning > > > > > > the > > > > > > > > > > second > > > > > > > > > > > > open > > > > > > > > > > > > >> > >> >> question > > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > > >> > >> >> > > > > > > setting > > > > > > > > > > > > >> > >> >> > > > > > > > > or > > > > > > > > > > > > >> > >> >> > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > setting > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > memory > > > > > > > > > > limit, I > > > > > > > > > > > > >> would > > > > > > > > > > > > >> > >> also > > > > > > > > > > > > >> > >> >> be > > > > > > > > > > > > >> > >> >> > > > > > interested > > > > > > > > > > > > >> > >> >> > > > > > > > why > > > > > > > > > > > > >> > >> >> > > > > > > > > > > Yang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Wang > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > thinks > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > leaving > > > > it > > > > > > > open > > > > > > > > > > would > > > > > > > > > > > be > > > > > > > > > > > > >> > best. > > > > > > > > > > > > >> > >> My > > > > > > > > > > > > >> > >> >> > > concern > > > > > > > > > > > > >> > >> >> > > > > > about > > > > > > > > > > > > >> > >> >> > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > be > > in > > > a > > > > > > > similar > > > > > > > > > > > > situation > > > > > > > > > > > > >> as > > > > > > > > > > > > >> > we > > > > > > > > > > > > >> > >> >> are > > > > > > > > > > > > >> > >> >> > now > > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > RocksDBStateBackend. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > If > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > the > > > > > > different > > > > > > > > > memory > > > > > > > > > > > > pools > > > > > > > > > > > > >> > are > > > > > > > > > > > > >> > >> not > > > > > > > > > > > > >> > >> >> > > > clearly > > > > > > > > > > > > >> > >> >> > > > > > > > > separated > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > can > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > spill > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > over > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > a > > > > > different > > > > > > > > pool, > > > > > > > > > > then > > > > > > > > > > > > it > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > >> quite > > > > > > > > > > > > >> > >> >> > hard > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > understand > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > what > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > exactly > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > causes a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > process > > > > to > > > > > > get > > > > > > > > > > killed > > > > > > > > > > > > for > > > > > > > > > > > > >> > using > > > > > > > > > > > > >> > >> >> too > > > > > > > > > > > > >> > >> >> > > much > > > > > > > > > > > > >> > >> >> > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > This > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > then > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > easily > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > lead > > > to > > > > a > > > > > > > > similar > > > > > > > > > > > > >> situation > > > > > > > > > > > > >> > >> what > > > > > > > > > > > > >> > >> >> we > > > > > > > > > > > > >> > >> >> > > have > > > > > > > > > > > > >> > >> >> > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > cutoff-ratio. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > So > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > why > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > setting > > > > a > > > > > > sane > > > > > > > > > > default > > > > > > > > > > > > >> value > > > > > > > > > > > > >> > >> for > > > > > > > > > > > > >> > >> >> max > > > > > > > > > > > > >> > >> >> > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > giving > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > option > > > > to > > > > > > > > increase > > > > > > > > > > it > > > > > > > > > > > if > > > > > > > > > > > > >> he > > > > > > > > > > > > >> > >> runs > > > > > > > > > > > > >> > >> >> into > > > > > > > > > > > > >> > >> >> > > an > > > > > > > > > > > > >> > >> >> > > > > OOM. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > @Xintong, > > > > > > how > > > > > > > > > would > > > > > > > > > > > > >> > >> alternative 2 > > > > > > > > > > > > >> > >> >> > lead > > > > > > > > > > > > >> > >> >> > > to > > > > > > > > > > > > >> > >> >> > > > > > lower > > > > > > > > > > > > >> > >> >> > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > utilization > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > than > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > alternative > > > > > > 3 > > > > > > > > > where > > > > > > > > > > we > > > > > > > > > > > > set > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> direct > > > > > > > > > > > > >> > >> >> > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > to a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > higher > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > value? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > Till > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > On > > > Fri, > > > > > Aug > > > > > > 9, > > > > > > > > > 2019 > > > > > > > > > > at > > > > > > > > > > > > >> 9:12 > > > > > > > > > > > > >> > AM > > > > > > > > > > > > >> > >> >> > Xintong > > > > > > > > > > > > >> > >> >> > > > > Song < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > for > > > > > > > the > > > > > > > > > > > > feedback, > > > > > > > > > > > > >> > >> Yang. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Regarding > > > > > > > your > > > > > > > > > > > > comments: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > *Native > > > > > > and > > > > > > > > > Direct > > > > > > > > > > > > >> Memory* > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > I > > > > think > > > > > > > > setting > > > > > > > > > a > > > > > > > > > > > very > > > > > > > > > > > > >> > large > > > > > > > > > > > > >> > >> max > > > > > > > > > > > > >> > >> >> > > direct > > > > > > > > > > > > >> > >> >> > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > size > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > definitely > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > has > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > some > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > good > > > > > > sides. > > > > > > > > > E.g., > > > > > > > > > > we > > > > > > > > > > > > do > > > > > > > > > > > > >> not > > > > > > > > > > > > >> > >> >> worry > > > > > > > > > > > > >> > >> >> > > about > > > > > > > > > > > > >> > >> >> > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > OOM, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > don't > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > even > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > need > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > to > > > > > > allocate > > > > > > > > > > managed > > > > > > > > > > > / > > > > > > > > > > > > >> > network > > > > > > > > > > > > >> > >> >> > memory > > > > > > > > > > > > >> > >> >> > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > Unsafe.allocate() > > . > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > However, > > > > > > > there > > > > > > > > > are > > > > > > > > > > > > also > > > > > > > > > > > > >> > some > > > > > > > > > > > > >> > >> >> down > > > > > > > > > > > > >> > >> >> > > sides > > > > > > > > > > > > >> > >> >> > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > doing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > this. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > One > > > > > > > > thing I > > > > > > > > > > can > > > > > > > > > > > > >> think > > > > > > > > > > > > >> > >> of is > > > > > > > > > > > > >> > >> >> > that > > > > > > > > > > > > >> > >> >> > > > if > > > > > > > > > > > > >> > >> >> > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > task > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > executor > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > container > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > killed > > > > > > > due > > > > > > > > to > > > > > > > > > > > > >> overusing > > > > > > > > > > > > >> > >> >> memory, > > > > > > > > > > > > >> > >> >> > it > > > > > > > > > > > > >> > >> >> > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > hard > > > > > > > > > > > > >> > >> >> > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > use > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > know > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > part > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > the > > > > > > > > memory > > > > > > > > > > is > > > > > > > > > > > > >> > overused. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > Another > > > > > > > > > down > > > > > > > > > > > side > > > > > > > > > > > > >> is > > > > > > > > > > > > >> > >> that > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > JVM > > > > > > > > > > > > >> > >> >> > > > > > never > > > > > > > > > > > > >> > >> >> > > > > > > > > > trigger > > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > due > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > reaching > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > max > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > direct > > > > > > > > memory > > > > > > > > > > > > limit, > > > > > > > > > > > > >> > >> because > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > limit > > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > too > > > > > > > > > > > > >> > >> >> > > > > > > > > > high > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > reached. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > That > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > means > > > > > > we > > > > > > > > kind > > > > > > > > > > of > > > > > > > > > > > > >> relay > > > > > > > > > > > > >> > on > > > > > > > > > > > > >> > >> >> heap > > > > > > > > > > > > >> > >> >> > > > memory > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > trigger > > > > > > > > > > > > >> > >> >> > > > > > > > > > > GC > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > release > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > memory. > > > > > > > > That > > > > > > > > > > > could > > > > > > > > > > > > >> be a > > > > > > > > > > > > >> > >> >> problem > > > > > > > > > > > > >> > >> >> > in > > > > > > > > > > > > >> > >> >> > > > > cases > > > > > > > > > > > > >> > >> >> > > > > > > > where > > > > > > > > > > > > >> > >> >> > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > more > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > direct > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > usage > > > > > > but > > > > > > > > not > > > > > > > > > > > > enough > > > > > > > > > > > > >> > heap > > > > > > > > > > > > >> > >> >> > activity > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > trigger > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > GC. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Maybe > > > > > you > > > > > > > can > > > > > > > > > > share > > > > > > > > > > > > your > > > > > > > > > > > > >> > >> reasons > > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > > >> > >> >> > > > > > > preferring > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > setting a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > very > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > value, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > if > > > > there > > > > > > are > > > > > > > > > > > anything > > > > > > > > > > > > >> else > > > > > > > > > > > > >> > I > > > > > > > > > > > > >> > >> >> > > > overlooked. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > *Memory > > > > > > > > > > Calculation* > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > If > > > > there > > > > > > is > > > > > > > > any > > > > > > > > > > > > conflict > > > > > > > > > > > > >> > >> between > > > > > > > > > > > > >> > >> >> > > > multiple > > > > > > > > > > > > >> > >> >> > > > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > user > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > explicitly > > > > > > > > > > > specified, > > > > > > > > > > > > I > > > > > > > > > > > > >> > >> think we > > > > > > > > > > > > >> > >> >> > > should > > > > > > > > > > > > >> > >> >> > > > > > throw > > > > > > > > > > > > >> > >> >> > > > > > > > an > > > > > > > > > > > > >> > >> >> > > > > > > > > > > error. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > I > > > > think > > > > > > > doing > > > > > > > > > > > checking > > > > > > > > > > > > >> on > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > client > > > > > > > > > > > > >> > >> >> > > > side > > > > > > > > > > > > >> > >> >> > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > a > > > > > > > > > > > > >> > >> >> > > > > > > > > good > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > idea, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > so > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > that > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Yarn / > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > K8s > > > we > > > > > can > > > > > > > > > > discover > > > > > > > > > > > > the > > > > > > > > > > > > >> > >> problem > > > > > > > > > > > > >> > >> >> > > before > > > > > > > > > > > > >> > >> >> > > > > > > > submitting > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > Flink > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > cluster, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > which > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > is > > > > > always > > > > > > a > > > > > > > > good > > > > > > > > > > > > thing. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > But > > > we > > > > > can > > > > > > > not > > > > > > > > > > only > > > > > > > > > > > > >> rely on > > > > > > > > > > > > >> > >> the > > > > > > > > > > > > >> > >> >> > > client > > > > > > > > > > > > >> > >> >> > > > > side > > > > > > > > > > > > >> > >> >> > > > > > > > > > checking, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > because > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > standalone > > > > > > > > > cluster > > > > > > > > > > > > >> > >> TaskManagers > > > > > > > > > > > > >> > >> >> on > > > > > > > > > > > > >> > >> >> > > > > > different > > > > > > > > > > > > >> > >> >> > > > > > > > > > machines > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > may > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > have > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > configurations > > > > > > > > > and > > > > > > > > > > > the > > > > > > > > > > > > >> > client > > > > > > > > > > > > >> > >> >> does > > > > > > > > > > > > >> > >> >> > > see > > > > > > > > > > > > >> > >> >> > > > > > that. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > What > > > > do > > > > > > you > > > > > > > > > think? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > On > > > > Thu, > > > > > > Aug > > > > > > > 8, > > > > > > > > > > 2019 > > > > > > > > > > > at > > > > > > > > > > > > >> 5:09 > > > > > > > > > > > > >> > >> PM > > > > > > > > > > > > >> > >> >> Yang > > > > > > > > > > > > >> > >> >> > > > Wang > > > > > > > > > > > > >> > >> >> > > > > < > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > xintong, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > for > > > > > > > > > your > > > > > > > > > > > > >> detailed > > > > > > > > > > > > >> > >> >> > proposal. > > > > > > > > > > > > >> > >> >> > > > > After > > > > > > > > > > > > >> > >> >> > > > > > > all > > > > > > > > > > > > >> > >> >> > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > introduced, > > > > > > > > it > > > > > > > > > > > will > > > > > > > > > > > > be > > > > > > > > > > > > >> > more > > > > > > > > > > > > >> > >> >> > > powerful > > > > > > > > > > > > >> > >> >> > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > control > > > > > > > > > > > > >> > >> >> > > > > > > > > > the > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > flink > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > usage. I > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > just > > > > > > have > > > > > > > > few > > > > > > > > > > > > >> questions > > > > > > > > > > > > >> > >> about > > > > > > > > > > > > >> > >> >> it. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Native > > > > > > > > > and > > > > > > > > > > > > Direct > > > > > > > > > > > > >> > >> Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > do > > > > > > not > > > > > > > > > > > > >> differentiate > > > > > > > > > > > > >> > >> user > > > > > > > > > > > > >> > >> >> > direct > > > > > > > > > > > > >> > >> >> > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > native > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > memory. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > They > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > are > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > all > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > included > > > > > > > in > > > > > > > > > task > > > > > > > > > > > > >> off-heap > > > > > > > > > > > > >> > >> >> memory. > > > > > > > > > > > > >> > >> >> > > > > Right? > > > > > > > > > > > > >> > >> >> > > > > > > So i > > > > > > > > > > > > >> > >> >> > > > > > > > > > don’t > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > think > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > could > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > not > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > set > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > >> > -XX:MaxDirectMemorySize > > > > > > > > > > > > >> > >> >> > > > properly. I > > > > > > > > > > > > >> > >> >> > > > > > > > prefer > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > leaving > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > it a > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > very > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > large > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > value. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > Memory > > > > > > > > > > > > >> Calculation > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > If > > > > the > > > > > > sum > > > > > > > > of > > > > > > > > > > and > > > > > > > > > > > > >> > >> fine-grained > > > > > > > > > > > > >> > >> >> > > > > > > memory(network > > > > > > > > > > > > >> > >> >> > > > > > > > > > > memory, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > managed > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > memory, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > etc.) > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > is > > > > > > larger > > > > > > > > than > > > > > > > > > > > total > > > > > > > > > > > > >> > >> process > > > > > > > > > > > > >> > >> >> > > memory, > > > > > > > > > > > > >> > >> >> > > > > how > > > > > > > > > > > > >> > >> >> > > > > > do > > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > deal > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > with > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > situation? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Do > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > we > > > > > need > > > > > > to > > > > > > > > > check > > > > > > > > > > > the > > > > > > > > > > > > >> > memory > > > > > > > > > > > > >> > >> >> > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > client? > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > Song < > > > > > > > > > > > > >> > >> >> > > [hidden email]> > > > > > > > > > > > > >> > >> >> > > > > > > > > > 于2019年8月7日周三 > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > 下午10:14写道： > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi > > > > > > > > everyone, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We > > > > > > would > > > > > > > > > like > > > > > > > > > > to > > > > > > > > > > > > >> start > > > > > > > > > > > > >> > a > > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > > >> > >> >> > > > > > > thread > > > > > > > > > > > > >> > >> >> > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > "FLIP-49: > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > Unified > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > Memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Configuration > > > > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > TaskExecutors"[1], > > > > > > > > > > > > >> > >> >> > > > > > > where > > > > > > > > > > > > >> > >> >> > > > > > > > we > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > describe > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > how > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > improve > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > configurations. > > > > > > > > > > > > >> > >> >> > > > > The > > > > > > > > > > > > >> > >> >> > > > > > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > document > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > mostly > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > based > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > on > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > an > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > early > > > > > > > > design > > > > > > > > > > > > "Memory > > > > > > > > > > > > >> > >> >> Management > > > > > > > > > > > > >> > >> >> > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > Configuration > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > Reloaded"[2] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Stephan, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > updates > > > > > > > > > > > from > > > > > > > > > > > > >> > >> follow-up > > > > > > > > > > > > >> > >> >> > > > > discussions > > > > > > > > > > > > >> > >> >> > > > > > > > both > > > > > > > > > > > > >> > >> >> > > > > > > > > > > online > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > offline. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This > > > > > > > FLIP > > > > > > > > > > > > addresses > > > > > > > > > > > > >> > >> several > > > > > > > > > > > > >> > >> >> > > > > > shortcomings > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > current > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > (Flink > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > 1.9) > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TaskExecutor > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > configuration. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > Different > > > > > > > > > > > > >> > >> configuration > > > > > > > > > > > > >> > >> >> > for > > > > > > > > > > > > >> > >> >> > > > > > > Streaming > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > Batch. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Complex > > > > > > > > > > and > > > > > > > > > > > > >> > >> difficult > > > > > > > > > > > > >> > >> >> > > > > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > RocksDB > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > in > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > Streaming. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > Complicated, > > > > > > > > > > > > >> > >> uncertain > > > > > > > > > > > > >> > >> >> and > > > > > > > > > > > > >> > >> >> > > > hard > > > > > > > > > > > > >> > >> >> > > > > to > > > > > > > > > > > > >> > >> >> > > > > > > > > > > understand. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Key > > > > > > > > changes > > > > > > > > > to > > > > > > > > > > > > solve > > > > > > > > > > > > >> > the > > > > > > > > > > > > >> > >> >> > problems > > > > > > > > > > > > >> > >> >> > > > can > > > > > > > > > > > > >> > >> >> > > > > > be > > > > > > > > > > > > >> > >> >> > > > > > > > > > > summarized > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > as > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > follows. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Extend > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> manager > > > > > > > > > > > > >> > >> >> to > > > > > > > > > > > > >> > >> >> > > also > > > > > > > > > > > > >> > >> >> > > > > > > account > > > > > > > > > > > > >> > >> >> > > > > > > > > for > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > usage > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > by > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > backends. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > Modify > > > > > > > > > > how > > > > > > > > > > > > >> > >> TaskExecutor > > > > > > > > > > > > >> > >> >> > > memory > > > > > > > > > > > > >> > >> >> > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > partitioned > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > accounted > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > individual > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > memory > > > > > > > > > > > > >> reservations > > > > > > > > > > > > >> > >> and > > > > > > > > > > > > >> > >> >> > pools. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > Simplify > > > > > > > > > > > > memory > > > > > > > > > > > > >> > >> >> > > configuration > > > > > > > > > > > > >> > >> >> > > > > > > options > > > > > > > > > > > > >> > >> >> > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > calculations > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > logics. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please > > > > > > > > find > > > > > > > > > > more > > > > > > > > > > > > >> > details > > > > > > > > > > > > >> > >> in > > > > > > > > > > > > >> > >> >> the > > > > > > > > > > > > >> > >> >> > > > FLIP > > > > > > > > > > > > >> > >> >> > > > > > wiki > > > > > > > > > > > > >> > >> >> > > > > > > > > > > document > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > [1]. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (Please > > > > > > > > note > > > > > > > > > > > that > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> early > > > > > > > > > > > > >> > >> >> > > design > > > > > > > > > > > > >> > >> >> > > > > doc > > > > > > > > > > > > >> > >> >> > > > > > > [2] > > > > > > > > > > > > >> > >> >> > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > out > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > of > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > sync, > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > and > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated > > > > > > > > > to > > > > > > > > > > > > have > > > > > > > > > > > > >> the > > > > > > > > > > > > >> > >> >> > > discussion > > > > > > > > > > > > >> > >> >> > > > in > > > > > > > > > > > > >> > >> >> > > > > > > this > > > > > > > > > > > > >> > >> >> > > > > > > > > > > mailing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > list > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > thread.) > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Looking > > > > > > > > > > forward > > > > > > > > > > > to > > > > > > > > > > > > >> your > > > > > > > > > > > > >> > >> >> > > feedbacks. > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank > > > > > > > you~ > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > Song > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [2] > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1o4KvyyXsQMGUastfPin3ZWeUXWsJgoL7piqp1fFYJvA/edit?usp=sharing > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > > >> > >> >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |