[DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
Locked
12
12
 
|
Hi all
Users report they meet serious memory leak when submitting jobs continously in session mode within k8s (please refer to FLINK-18712[1] ), and I also reproduce this to find this is caused by memory fragmentation of glibc [2][3] and provide solutions to fix this: * Quick but not very clean solution to limit the memory pool of glibc, limit MALLOC_ARENA_MAX to 2 * More general solution by rebuilding the image to install libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD The reporter adopted the 2nd solution to fix this issue eventually. Thus, I begin to think whether we should change our Dockerfile to adopt jemalloc as default memory allocator [4]. From my point of view, we have two choices: 1. Introduce another Dockerfile using jemalloc as default memory allocator, which means Flink needs another two new image tags to build docker with jemalloc while default docker still use glibc. 2. Set the default memory allocator as jemalloc in our existing Dockerfiles, which means Flink offer docker image with jemalloc by default. I prefer the 2nd option as our company already use jemalloc as default memory allocator for JDK at our production environment due to messages from os team warning of glibc's memory fragmentation. Moreover, I found several open source projects adopting jemalloc as default memory allocator within their images to resolve memory fragmentation problem, e.g fluent [5], home-assistant [6]. What do you guys think of this issue? [1] https://issues.apache.org/jira/browse/FLINK-18712 [2] https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 [4] https://issues.apache.org/jira/browse/FLINK-19125 [5] https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux [6] https://github.com/home-assistant/core/pull/33237 Best Yun Tang |
|
|
Thanks for debugging and resolving the issue and driving the discussion Yun!
For the given solutions, I prefer option 1 (supply another Dockerfile using jemalloc as default memory allocator) because of the below reasons: 1. It's hard to say jemalloc is always better than ptmalloc (glibc malloc), or else glibc should have already adopted it as the default memory allocator. And as indicated here [1], in some cases jemalloc will consume as much as twice the memory than glibc 2. All existing Flink docker images use glibc, if we change the default memory allocator to jemalloc and only supply one series of images, we will leave those having better performance with glibc no other choices but staying with old images. In another word, there's a risk of introducing new problems while fixing an existing one if choosing option-2. And there is a third option considering the efforts of maintaining more images if the memory leak issue is not widely observed, that we could document the steps of building Dockerfile with jemalloc as default allocator so users could build it when needed, which leaves the burden to our users so for me it's not the best option. Best Regards, Yu [1] https://stackoverflow.com/a/33993215 On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > Hi all > > Users report they meet serious memory leak when submitting jobs > continously in session mode within k8s (please refer to FLINK-18712[1] ), > and I also reproduce this to find this is caused by memory fragmentation of > glibc [2][3] and provide solutions to fix this: > > * Quick but not very clean solution to limit the memory pool of glibc, > limit MALLOC_ARENA_MAX to 2 > > * More general solution by rebuilding the image to install > libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > The reporter adopted the 2nd solution to fix this issue eventually. Thus, > I begin to think whether we should change our Dockerfile to adopt jemalloc > as default memory allocator [4]. > From my point of view, we have two choices: > > 1. Introduce another Dockerfile using jemalloc as default memory > allocator, which means Flink needs another two new image tags to build > docker with jemalloc while default docker still use glibc. > 2. Set the default memory allocator as jemalloc in our existing > Dockerfiles, which means Flink offer docker image with jemalloc by default. > > I prefer the 2nd option as our company already use jemalloc as default > memory allocator for JDK at our production environment due to messages from > os team warning of glibc's memory fragmentation. > Moreover, I found several open source projects adopting jemalloc as > default memory allocator within their images to resolve memory > fragmentation problem, e.g fluent [5], home-assistant [6]. > > What do you guys think of this issue? > > [1] https://issues.apache.org/jira/browse/FLINK-18712 > [2] > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > [4] https://issues.apache.org/jira/browse/FLINK-19125 > [5] > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > [6] https://github.com/home-assistant/core/pull/33237 > > > Best > Yun Tang > |
|
|
Thanks Yun Tang for starting this discussion.
I think this is very important when deploy Flink with container environment in production. I just have quick question. Could we have both memory allocator(e.g. glibc, jemalloc) in the Flink official image and enable a specific one by setting ENV? Best, Yang Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > Thanks for debugging and resolving the issue and driving the discussion > Yun! > > For the given solutions, I prefer option 1 (supply another Dockerfile using > jemalloc as default memory allocator) because of the below reasons: > > 1. It's hard to say jemalloc is always better than ptmalloc (glibc malloc), > or else glibc should have already adopted it as the default memory > allocator. And as indicated here [1], in some cases jemalloc will > consume as much as twice the memory than glibc > > 2. All existing Flink docker images use glibc, if we change the default > memory allocator to jemalloc and only supply one series of images, we will > leave those having better performance with glibc no other choices but > staying with old images. In another word, there's a risk of introducing new > problems while fixing an existing one if choosing option-2. > > And there is a third option considering the efforts of maintaining more > images if the memory leak issue is not widely observed, that we could > document the steps of building Dockerfile with jemalloc as default > allocator so users could build it when needed, which leaves the burden to > our users so for me it's not the best option. > > Best Regards, > Yu > > [1] https://stackoverflow.com/a/33993215 > > On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > > > Hi all > > > > Users report they meet serious memory leak when submitting jobs > > continously in session mode within k8s (please refer to FLINK-18712[1] ), > > and I also reproduce this to find this is caused by memory fragmentation > of > > glibc [2][3] and provide solutions to fix this: > > > > * Quick but not very clean solution to limit the memory pool of > glibc, > > limit MALLOC_ARENA_MAX to 2 > > > > * More general solution by rebuilding the image to install > > libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > The reporter adopted the 2nd solution to fix this issue eventually. Thus, > > I begin to think whether we should change our Dockerfile to adopt > jemalloc > > as default memory allocator [4]. > > From my point of view, we have two choices: > > > > 1. Introduce another Dockerfile using jemalloc as default memory > > allocator, which means Flink needs another two new image tags to build > > docker with jemalloc while default docker still use glibc. > > 2. Set the default memory allocator as jemalloc in our existing > > Dockerfiles, which means Flink offer docker image with jemalloc by > default. > > > > I prefer the 2nd option as our company already use jemalloc as default > > memory allocator for JDK at our production environment due to messages > from > > os team warning of glibc's memory fragmentation. > > Moreover, I found several open source projects adopting jemalloc as > > default memory allocator within their images to resolve memory > > fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > What do you guys think of this issue? > > > > [1] https://issues.apache.org/jira/browse/FLINK-18712 > > [2] > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > [4] https://issues.apache.org/jira/browse/FLINK-19125 > > [5] > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > [6] https://github.com/home-assistant/core/pull/33237 > > > > > > Best > > Yun Tang > > > |
|
|
Thanks for Yang's suggestion. I think this could be a better choice.
We could install jemalloc and only enable it in LD_PRELOAD when user pass specific configuration for docker-entrypoint.sh. By doing so, we could avoid to create another docker image tags and also offer ability to reduce memory fragmentation problem. Does anyone else have other ideas? Best Yun Tang ________________________________ From: Yang Wang <[hidden email]> Sent: Thursday, October 15, 2020 14:59 To: dev <[hidden email]> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Thanks Yun Tang for starting this discussion. I think this is very important when deploy Flink with container environment in production. I just have quick question. Could we have both memory allocator(e.g. glibc, jemalloc) in the Flink official image and enable a specific one by setting ENV? Best, Yang Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > Thanks for debugging and resolving the issue and driving the discussion > Yun! > > For the given solutions, I prefer option 1 (supply another Dockerfile using > jemalloc as default memory allocator) because of the below reasons: > > 1. It's hard to say jemalloc is always better than ptmalloc (glibc malloc), > or else glibc should have already adopted it as the default memory > allocator. And as indicated here [1], in some cases jemalloc will > consume as much as twice the memory than glibc > > 2. All existing Flink docker images use glibc, if we change the default > memory allocator to jemalloc and only supply one series of images, we will > leave those having better performance with glibc no other choices but > staying with old images. In another word, there's a risk of introducing new > problems while fixing an existing one if choosing option-2. > > And there is a third option considering the efforts of maintaining more > images if the memory leak issue is not widely observed, that we could > document the steps of building Dockerfile with jemalloc as default > allocator so users could build it when needed, which leaves the burden to > our users so for me it's not the best option. > > Best Regards, > Yu > > [1] https://stackoverflow.com/a/33993215 > > On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > > > Hi all > > > > Users report they meet serious memory leak when submitting jobs > > continously in session mode within k8s (please refer to FLINK-18712[1] ), > > and I also reproduce this to find this is caused by memory fragmentation > of > > glibc [2][3] and provide solutions to fix this: > > > > * Quick but not very clean solution to limit the memory pool of > glibc, > > limit MALLOC_ARENA_MAX to 2 > > > > * More general solution by rebuilding the image to install > > libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > The reporter adopted the 2nd solution to fix this issue eventually. Thus, > > I begin to think whether we should change our Dockerfile to adopt > jemalloc > > as default memory allocator [4]. > > From my point of view, we have two choices: > > > > 1. Introduce another Dockerfile using jemalloc as default memory > > allocator, which means Flink needs another two new image tags to build > > docker with jemalloc while default docker still use glibc. > > 2. Set the default memory allocator as jemalloc in our existing > > Dockerfiles, which means Flink offer docker image with jemalloc by > default. > > > > I prefer the 2nd option as our company already use jemalloc as default > > memory allocator for JDK at our production environment due to messages > from > > os team warning of glibc's memory fragmentation. > > Moreover, I found several open source projects adopting jemalloc as > > default memory allocator within their images to resolve memory > > fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > What do you guys think of this issue? > > > > [1] https://issues.apache.org/jira/browse/FLINK-18712 > > [2] > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > [4] https://issues.apache.org/jira/browse/FLINK-19125 > > [5] > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > [6] https://github.com/home-assistant/core/pull/33237 > > > > > > Best > > Yun Tang > > > |
Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
|
|
If it is possible to support both allocators in a single image then we
should definitely go with that option. On 10/16/2020 12:21 PM, Yun Tang wrote: > Thanks for Yang's suggestion. I think this could be a better choice. > We could install jemalloc and only enable it in LD_PRELOAD when user pass specific configuration for docker-entrypoint.sh. > By doing so, we could avoid to create another docker image tags and also offer ability to reduce memory fragmentation problem. > > Does anyone else have other ideas? > > Best > Yun Tang > ________________________________ > From: Yang Wang <[hidden email]> > Sent: Thursday, October 15, 2020 14:59 > To: dev <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image > > Thanks Yun Tang for starting this discussion. > > I think this is very important when deploy Flink with container environment > in production. I just > have quick question. Could we have both memory allocator(e.g. glibc, > jemalloc) in the Flink > official image and enable a specific one by setting ENV? > > Best, > Yang > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > >> Thanks for debugging and resolving the issue and driving the discussion >> Yun! >> >> For the given solutions, I prefer option 1 (supply another Dockerfile using >> jemalloc as default memory allocator) because of the below reasons: >> >> 1. It's hard to say jemalloc is always better than ptmalloc (glibc malloc), >> or else glibc should have already adopted it as the default memory >> allocator. And as indicated here [1], in some cases jemalloc will >> consume as much as twice the memory than glibc >> >> 2. All existing Flink docker images use glibc, if we change the default >> memory allocator to jemalloc and only supply one series of images, we will >> leave those having better performance with glibc no other choices but >> staying with old images. In another word, there's a risk of introducing new >> problems while fixing an existing one if choosing option-2. >> >> And there is a third option considering the efforts of maintaining more >> images if the memory leak issue is not widely observed, that we could >> document the steps of building Dockerfile with jemalloc as default >> allocator so users could build it when needed, which leaves the burden to >> our users so for me it's not the best option. >> >> Best Regards, >> Yu >> >> [1] https://stackoverflow.com/a/33993215 >> >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: >> >>> Hi all >>> >>> Users report they meet serious memory leak when submitting jobs >>> continously in session mode within k8s (please refer to FLINK-18712[1] ), >>> and I also reproduce this to find this is caused by memory fragmentation >> of >>> glibc [2][3] and provide solutions to fix this: >>> >>> * Quick but not very clean solution to limit the memory pool of >> glibc, >>> limit MALLOC_ARENA_MAX to 2 >>> >>> * More general solution by rebuilding the image to install >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD >>> >>> The reporter adopted the 2nd solution to fix this issue eventually. Thus, >>> I begin to think whether we should change our Dockerfile to adopt >> jemalloc >>> as default memory allocator [4]. >>> From my point of view, we have two choices: >>> >>> 1. Introduce another Dockerfile using jemalloc as default memory >>> allocator, which means Flink needs another two new image tags to build >>> docker with jemalloc while default docker still use glibc. >>> 2. Set the default memory allocator as jemalloc in our existing >>> Dockerfiles, which means Flink offer docker image with jemalloc by >> default. >>> I prefer the 2nd option as our company already use jemalloc as default >>> memory allocator for JDK at our production environment due to messages >> from >>> os team warning of glibc's memory fragmentation. >>> Moreover, I found several open source projects adopting jemalloc as >>> default memory allocator within their images to resolve memory >>> fragmentation problem, e.g fluent [5], home-assistant [6]. >>> >>> What do you guys think of this issue? >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 >>> [2] >>> >> https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 >>> [5] >>> >> https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux >>> [6] https://github.com/home-assistant/core/pull/33237 >>> >>> >>> Best >>> Yun Tang >>> |
Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
|
+1 for making it configurable in the same Docker image.
Cheers, Till On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler <[hidden email]> wrote: > If it is possible to support both allocators in a single image then we > should definitely go with that option. > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > Thanks for Yang's suggestion. I think this could be a better choice. > > We could install jemalloc and only enable it in LD_PRELOAD when user > pass specific configuration for docker-entrypoint.sh. > > By doing so, we could avoid to create another docker image tags and also > offer ability to reduce memory fragmentation problem. > > > > Does anyone else have other ideas? > > > > Best > > Yun Tang > > ________________________________ > > From: Yang Wang <[hidden email]> > > Sent: Thursday, October 15, 2020 14:59 > > To: dev <[hidden email]> > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > allocator for debian based Flink docker image > > > > Thanks Yun Tang for starting this discussion. > > > > I think this is very important when deploy Flink with container > environment > > in production. I just > > have quick question. Could we have both memory allocator(e.g. glibc, > > jemalloc) in the Flink > > official image and enable a specific one by setting ENV? > > > > Best, > > Yang > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > >> Thanks for debugging and resolving the issue and driving the discussion > >> Yun! > >> > >> For the given solutions, I prefer option 1 (supply another Dockerfile > using > >> jemalloc as default memory allocator) because of the below reasons: > >> > >> 1. It's hard to say jemalloc is always better than ptmalloc (glibc > malloc), > >> or else glibc should have already adopted it as the default memory > >> allocator. And as indicated here [1], in some cases jemalloc will > >> consume as much as twice the memory than glibc > >> > >> 2. All existing Flink docker images use glibc, if we change the default > >> memory allocator to jemalloc and only supply one series of images, we > will > >> leave those having better performance with glibc no other choices but > >> staying with old images. In another word, there's a risk of introducing > new > >> problems while fixing an existing one if choosing option-2. > >> > >> And there is a third option considering the efforts of maintaining more > >> images if the memory leak issue is not widely observed, that we could > >> document the steps of building Dockerfile with jemalloc as default > >> allocator so users could build it when needed, which leaves the burden > to > >> our users so for me it's not the best option. > >> > >> Best Regards, > >> Yu > >> > >> [1] https://stackoverflow.com/a/33993215 > >> > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > >> > >>> Hi all > >>> > >>> Users report they meet serious memory leak when submitting jobs > >>> continously in session mode within k8s (please refer to FLINK-18712[1] > ), > >>> and I also reproduce this to find this is caused by memory > fragmentation > >> of > >>> glibc [2][3] and provide solutions to fix this: > >>> > >>> * Quick but not very clean solution to limit the memory pool of > >> glibc, > >>> limit MALLOC_ARENA_MAX to 2 > >>> > >>> * More general solution by rebuilding the image to install > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > >>> > >>> The reporter adopted the 2nd solution to fix this issue eventually. > Thus, > >>> I begin to think whether we should change our Dockerfile to adopt > >> jemalloc > >>> as default memory allocator [4]. > >>> From my point of view, we have two choices: > >>> > >>> 1. Introduce another Dockerfile using jemalloc as default memory > >>> allocator, which means Flink needs another two new image tags to build > >>> docker with jemalloc while default docker still use glibc. > >>> 2. Set the default memory allocator as jemalloc in our existing > >>> Dockerfiles, which means Flink offer docker image with jemalloc by > >> default. > >>> I prefer the 2nd option as our company already use jemalloc as default > >>> memory allocator for JDK at our production environment due to messages > >> from > >>> os team warning of glibc's memory fragmentation. > >>> Moreover, I found several open source projects adopting jemalloc as > >>> default memory allocator within their images to resolve memory > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > >>> > >>> What do you guys think of this issue? > >>> > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > >>> [2] > >>> > >> > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > >>> [5] > >>> > >> > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > >>> [6] https://github.com/home-assistant/core/pull/33237 > >>> > >>> > >>> Best > >>> Yun Tang > >>> > > |
|
|
I'm also +1 on making it configurable in the same Docker image.
It seems we have reached consensus and there are already enough +1s to move forward, and suggest @Yun to conclude the discussion directly if there are no objections. Thanks. Best Regards, Yu On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> wrote: > +1 for making it configurable in the same Docker image. > > Cheers, > Till > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler <[hidden email]> > wrote: > > > If it is possible to support both allocators in a single image then we > > should definitely go with that option. > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > Thanks for Yang's suggestion. I think this could be a better choice. > > > We could install jemalloc and only enable it in LD_PRELOAD when user > > pass specific configuration for docker-entrypoint.sh. > > > By doing so, we could avoid to create another docker image tags and > also > > offer ability to reduce memory fragmentation problem. > > > > > > Does anyone else have other ideas? > > > > > > Best > > > Yun Tang > > > ________________________________ > > > From: Yang Wang <[hidden email]> > > > Sent: Thursday, October 15, 2020 14:59 > > > To: dev <[hidden email]> > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > allocator for debian based Flink docker image > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > I think this is very important when deploy Flink with container > > environment > > > in production. I just > > > have quick question. Could we have both memory allocator(e.g. glibc, > > > jemalloc) in the Flink > > > official image and enable a specific one by setting ENV? > > > > > > Best, > > > Yang > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > >> Thanks for debugging and resolving the issue and driving the > discussion > > >> Yun! > > >> > > >> For the given solutions, I prefer option 1 (supply another Dockerfile > > using > > >> jemalloc as default memory allocator) because of the below reasons: > > >> > > >> 1. It's hard to say jemalloc is always better than ptmalloc (glibc > > malloc), > > >> or else glibc should have already adopted it as the default memory > > >> allocator. And as indicated here [1], in some cases jemalloc will > > >> consume as much as twice the memory than glibc > > >> > > >> 2. All existing Flink docker images use glibc, if we change the > default > > >> memory allocator to jemalloc and only supply one series of images, we > > will > > >> leave those having better performance with glibc no other choices but > > >> staying with old images. In another word, there's a risk of > introducing > > new > > >> problems while fixing an existing one if choosing option-2. > > >> > > >> And there is a third option considering the efforts of maintaining > more > > >> images if the memory leak issue is not widely observed, that we could > > >> document the steps of building Dockerfile with jemalloc as default > > >> allocator so users could build it when needed, which leaves the burden > > to > > >> our users so for me it's not the best option. > > >> > > >> Best Regards, > > >> Yu > > >> > > >> [1] https://stackoverflow.com/a/33993215 > > >> > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > > >> > > >>> Hi all > > >>> > > >>> Users report they meet serious memory leak when submitting jobs > > >>> continously in session mode within k8s (please refer to > FLINK-18712[1] > > ), > > >>> and I also reproduce this to find this is caused by memory > > fragmentation > > >> of > > >>> glibc [2][3] and provide solutions to fix this: > > >>> > > >>> * Quick but not very clean solution to limit the memory pool of > > >> glibc, > > >>> limit MALLOC_ARENA_MAX to 2 > > >>> > > >>> * More general solution by rebuilding the image to install > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > >>> > > >>> The reporter adopted the 2nd solution to fix this issue eventually. > > Thus, > > >>> I begin to think whether we should change our Dockerfile to adopt > > >> jemalloc > > >>> as default memory allocator [4]. > > >>> From my point of view, we have two choices: > > >>> > > >>> 1. Introduce another Dockerfile using jemalloc as default memory > > >>> allocator, which means Flink needs another two new image tags to > build > > >>> docker with jemalloc while default docker still use glibc. > > >>> 2. Set the default memory allocator as jemalloc in our existing > > >>> Dockerfiles, which means Flink offer docker image with jemalloc by > > >> default. > > >>> I prefer the 2nd option as our company already use jemalloc as > default > > >>> memory allocator for JDK at our production environment due to > messages > > >> from > > >>> os team warning of glibc's memory fragmentation. > > >>> Moreover, I found several open source projects adopting jemalloc as > > >>> default memory allocator within their images to resolve memory > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > >>> > > >>> What do you guys think of this issue? > > >>> > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > >>> [2] > > >>> > > >> > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > >>> [5] > > >>> > > >> > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > >>> > > >>> > > >>> Best > > >>> Yun Tang > > >>> > > > > > |
Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
|
|
The only question left would be what will be the default value?
Cheers, Till On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]> wrote: > I'm also +1 on making it configurable in the same Docker image. > > It seems we have reached consensus and there are already enough +1s to move > forward, and suggest @Yun to conclude the discussion directly if there are > no objections. > > Thanks. > > Best Regards, > Yu > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> wrote: > > > +1 for making it configurable in the same Docker image. > > > > Cheers, > > Till > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler <[hidden email]> > > wrote: > > > > > If it is possible to support both allocators in a single image then we > > > should definitely go with that option. > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > Thanks for Yang's suggestion. I think this could be a better choice. > > > > We could install jemalloc and only enable it in LD_PRELOAD when user > > > pass specific configuration for docker-entrypoint.sh. > > > > By doing so, we could avoid to create another docker image tags and > > also > > > offer ability to reduce memory fragmentation problem. > > > > > > > > Does anyone else have other ideas? > > > > > > > > Best > > > > Yun Tang > > > > ________________________________ > > > > From: Yang Wang <[hidden email]> > > > > Sent: Thursday, October 15, 2020 14:59 > > > > To: dev <[hidden email]> > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > > allocator for debian based Flink docker image > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > I think this is very important when deploy Flink with container > > > environment > > > > in production. I just > > > > have quick question. Could we have both memory allocator(e.g. glibc, > > > > jemalloc) in the Flink > > > > official image and enable a specific one by setting ENV? > > > > > > > > Best, > > > > Yang > > > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > discussion > > > >> Yun! > > > >> > > > >> For the given solutions, I prefer option 1 (supply another > Dockerfile > > > using > > > >> jemalloc as default memory allocator) because of the below reasons: > > > >> > > > >> 1. It's hard to say jemalloc is always better than ptmalloc (glibc > > > malloc), > > > >> or else glibc should have already adopted it as the default memory > > > >> allocator. And as indicated here [1], in some cases jemalloc will > > > >> consume as much as twice the memory than glibc > > > >> > > > >> 2. All existing Flink docker images use glibc, if we change the > > default > > > >> memory allocator to jemalloc and only supply one series of images, > we > > > will > > > >> leave those having better performance with glibc no other choices > but > > > >> staying with old images. In another word, there's a risk of > > introducing > > > new > > > >> problems while fixing an existing one if choosing option-2. > > > >> > > > >> And there is a third option considering the efforts of maintaining > > more > > > >> images if the memory leak issue is not widely observed, that we > could > > > >> document the steps of building Dockerfile with jemalloc as default > > > >> allocator so users could build it when needed, which leaves the > burden > > > to > > > >> our users so for me it's not the best option. > > > >> > > > >> Best Regards, > > > >> Yu > > > >> > > > >> [1] https://stackoverflow.com/a/33993215 > > > >> > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > > > >> > > > >>> Hi all > > > >>> > > > >>> Users report they meet serious memory leak when submitting jobs > > > >>> continously in session mode within k8s (please refer to > > FLINK-18712[1] > > > ), > > > >>> and I also reproduce this to find this is caused by memory > > > fragmentation > > > >> of > > > >>> glibc [2][3] and provide solutions to fix this: > > > >>> > > > >>> * Quick but not very clean solution to limit the memory pool > of > > > >> glibc, > > > >>> limit MALLOC_ARENA_MAX to 2 > > > >>> > > > >>> * More general solution by rebuilding the image to install > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > >>> > > > >>> The reporter adopted the 2nd solution to fix this issue eventually. > > > Thus, > > > >>> I begin to think whether we should change our Dockerfile to adopt > > > >> jemalloc > > > >>> as default memory allocator [4]. > > > >>> From my point of view, we have two choices: > > > >>> > > > >>> 1. Introduce another Dockerfile using jemalloc as default > memory > > > >>> allocator, which means Flink needs another two new image tags to > > build > > > >>> docker with jemalloc while default docker still use glibc. > > > >>> 2. Set the default memory allocator as jemalloc in our existing > > > >>> Dockerfiles, which means Flink offer docker image with jemalloc by > > > >> default. > > > >>> I prefer the 2nd option as our company already use jemalloc as > > default > > > >>> memory allocator for JDK at our production environment due to > > messages > > > >> from > > > >>> os team warning of glibc's memory fragmentation. > > > >>> Moreover, I found several open source projects adopting jemalloc as > > > >>> default memory allocator within their images to resolve memory > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > >>> > > > >>> What do you guys think of this issue? > > > >>> > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > >>> [2] > > > >>> > > > >> > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > >>> [5] > > > >>> > > > >> > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > >>> > > > >>> > > > >>> Best > > > >>> Yun Tang > > > >>> > > > > > > > > > |
|
|
True, thanks for the reminder Till!
I suggest using glibc malloc as default. On one hand this follows our old behavior (only with glibc malloc support in the image), on the other hand I believe glibc isn't using jemalloc as its default memory allocator for some reason. Please let me know your thoughts. Thanks. Best Regards, Yu On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email]> wrote: > The only question left would be what will be the default value? > > Cheers, > Till > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]> wrote: > > > I'm also +1 on making it configurable in the same Docker image. > > > > It seems we have reached consensus and there are already enough +1s to > move > > forward, and suggest @Yun to conclude the discussion directly if there > are > > no objections. > > > > Thanks. > > > > Best Regards, > > Yu > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> > wrote: > > > > > +1 for making it configurable in the same Docker image. > > > > > > Cheers, > > > Till > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler <[hidden email]> > > > wrote: > > > > > > > If it is possible to support both allocators in a single image then > we > > > > should definitely go with that option. > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > Thanks for Yang's suggestion. I think this could be a better > choice. > > > > > We could install jemalloc and only enable it in LD_PRELOAD when > user > > > > pass specific configuration for docker-entrypoint.sh. > > > > > By doing so, we could avoid to create another docker image tags and > > > also > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > Best > > > > > Yun Tang > > > > > ________________________________ > > > > > From: Yang Wang <[hidden email]> > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > To: dev <[hidden email]> > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > > > allocator for debian based Flink docker image > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > I think this is very important when deploy Flink with container > > > > environment > > > > > in production. I just > > > > > have quick question. Could we have both memory allocator(e.g. > glibc, > > > > > jemalloc) in the Flink > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > Best, > > > > > Yang > > > > > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > discussion > > > > >> Yun! > > > > >> > > > > >> For the given solutions, I prefer option 1 (supply another > > Dockerfile > > > > using > > > > >> jemalloc as default memory allocator) because of the below > reasons: > > > > >> > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc (glibc > > > > malloc), > > > > >> or else glibc should have already adopted it as the default memory > > > > >> allocator. And as indicated here [1], in some cases jemalloc will > > > > >> consume as much as twice the memory than glibc > > > > >> > > > > >> 2. All existing Flink docker images use glibc, if we change the > > > default > > > > >> memory allocator to jemalloc and only supply one series of images, > > we > > > > will > > > > >> leave those having better performance with glibc no other choices > > but > > > > >> staying with old images. In another word, there's a risk of > > > introducing > > > > new > > > > >> problems while fixing an existing one if choosing option-2. > > > > >> > > > > >> And there is a third option considering the efforts of maintaining > > > more > > > > >> images if the memory leak issue is not widely observed, that we > > could > > > > >> document the steps of building Dockerfile with jemalloc as default > > > > >> allocator so users could build it when needed, which leaves the > > burden > > > > to > > > > >> our users so for me it's not the best option. > > > > >> > > > > >> Best Regards, > > > > >> Yu > > > > >> > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > >> > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> wrote: > > > > >> > > > > >>> Hi all > > > > >>> > > > > >>> Users report they meet serious memory leak when submitting jobs > > > > >>> continously in session mode within k8s (please refer to > > > FLINK-18712[1] > > > > ), > > > > >>> and I also reproduce this to find this is caused by memory > > > > fragmentation > > > > >> of > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > >>> > > > > >>> * Quick but not very clean solution to limit the memory pool > > of > > > > >> glibc, > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > >>> > > > > >>> * More general solution by rebuilding the image to install > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > >>> > > > > >>> The reporter adopted the 2nd solution to fix this issue > eventually. > > > > Thus, > > > > >>> I begin to think whether we should change our Dockerfile to adopt > > > > >> jemalloc > > > > >>> as default memory allocator [4]. > > > > >>> From my point of view, we have two choices: > > > > >>> > > > > >>> 1. Introduce another Dockerfile using jemalloc as default > > memory > > > > >>> allocator, which means Flink needs another two new image tags to > > > build > > > > >>> docker with jemalloc while default docker still use glibc. > > > > >>> 2. Set the default memory allocator as jemalloc in our > existing > > > > >>> Dockerfiles, which means Flink offer docker image with jemalloc > by > > > > >> default. > > > > >>> I prefer the 2nd option as our company already use jemalloc as > > > default > > > > >>> memory allocator for JDK at our production environment due to > > > messages > > > > >> from > > > > >>> os team warning of glibc's memory fragmentation. > > > > >>> Moreover, I found several open source projects adopting jemalloc > as > > > > >>> default memory allocator within their images to resolve memory > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > >>> > > > > >>> What do you guys think of this issue? > > > > >>> > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > >>> [2] > > > > >>> > > > > >> > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > >>> [5] > > > > >>> > > > > >> > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > >>> > > > > >>> > > > > >>> Best > > > > >>> Yun Tang > > > > >>> > > > > > > > > > > > > > > |
Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
|
|
Hi Yu,
I see your point why you are in favour of continuing using glibc. I honestly don't have a lot of experience with malloc libraries so I can only argue from a general perspective: We know that glibc has some problems wrt to memory fragmentation which can cause processes to exceed its memory limit. jemalloc seems to address this problem by avoiding memory fragmentation. However, it might be the case that jemalloc introduces other problems. In order to figure this out we would need to collect more data points/experience. So in the end we have to weigh glibc memory fragmentation against unknown problems in jemalloc. Given that choosing the malloc library will be quite an expert setting, I don't expect many Flink users to use it in case they see OOMs. Hence, I would be slightly in favour of choosing jemalloc as the default because it would allow us to gather feedback from users and solve the existing problem. In the best case, user's won't run into new problems. If they do, then we will have to re-evaluate this decision and might have to switch the malloc library back. W/o setting jemalloc as the default, I fear that we will never gather enough feedback to make jemalloc confidently the default. What one could argue, though, is that making jemalloc the default so shortly before the feature freeze limits the timespan we can observe how it behaves before the actual release. Doing such a change at the beginning of a release cycle, would allow us to gain more confidence. In any way, I believe that we should add a big release note and document what users should do in case they see memory issues (OOM kills, slower performance, etc.). Cheers, Till On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]> wrote: > True, thanks for the reminder Till! > > I suggest using glibc malloc as default. On one hand this follows our old > behavior (only with glibc malloc support in the image), on the other hand I > believe glibc isn't using jemalloc as its default memory allocator for some > reason. > > Please let me know your thoughts. Thanks. > > Best Regards, > Yu > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email]> wrote: > > > The only question left would be what will be the default value? > > > > Cheers, > > Till > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]> wrote: > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > It seems we have reached consensus and there are already enough +1s to > > move > > > forward, and suggest @Yun to conclude the discussion directly if there > > are > > > no objections. > > > > > > Thanks. > > > > > > Best Regards, > > > Yu > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> > > wrote: > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > [hidden email]> > > > > wrote: > > > > > > > > > If it is possible to support both allocators in a single image then > > we > > > > > should definitely go with that option. > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > Thanks for Yang's suggestion. I think this could be a better > > choice. > > > > > > We could install jemalloc and only enable it in LD_PRELOAD when > > user > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > By doing so, we could avoid to create another docker image tags > and > > > > also > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > Best > > > > > > Yun Tang > > > > > > ________________________________ > > > > > > From: Yang Wang <[hidden email]> > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > To: dev <[hidden email]> > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > I think this is very important when deploy Flink with container > > > > > environment > > > > > > in production. I just > > > > > > have quick question. Could we have both memory allocator(e.g. > > glibc, > > > > > > jemalloc) in the Flink > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > Best, > > > > > > Yang > > > > > > > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > > discussion > > > > > >> Yun! > > > > > >> > > > > > >> For the given solutions, I prefer option 1 (supply another > > > Dockerfile > > > > > using > > > > > >> jemalloc as default memory allocator) because of the below > > reasons: > > > > > >> > > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc > (glibc > > > > > malloc), > > > > > >> or else glibc should have already adopted it as the default > memory > > > > > >> allocator. And as indicated here [1], in some cases jemalloc > will > > > > > >> consume as much as twice the memory than glibc > > > > > >> > > > > > >> 2. All existing Flink docker images use glibc, if we change the > > > > default > > > > > >> memory allocator to jemalloc and only supply one series of > images, > > > we > > > > > will > > > > > >> leave those having better performance with glibc no other > choices > > > but > > > > > >> staying with old images. In another word, there's a risk of > > > > introducing > > > > > new > > > > > >> problems while fixing an existing one if choosing option-2. > > > > > >> > > > > > >> And there is a third option considering the efforts of > maintaining > > > > more > > > > > >> images if the memory leak issue is not widely observed, that we > > > could > > > > > >> document the steps of building Dockerfile with jemalloc as > default > > > > > >> allocator so users could build it when needed, which leaves the > > > burden > > > > > to > > > > > >> our users so for me it's not the best option. > > > > > >> > > > > > >> Best Regards, > > > > > >> Yu > > > > > >> > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > >> > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> > wrote: > > > > > >> > > > > > >>> Hi all > > > > > >>> > > > > > >>> Users report they meet serious memory leak when submitting jobs > > > > > >>> continously in session mode within k8s (please refer to > > > > FLINK-18712[1] > > > > > ), > > > > > >>> and I also reproduce this to find this is caused by memory > > > > > fragmentation > > > > > >> of > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > >>> > > > > > >>> * Quick but not very clean solution to limit the memory > pool > > > of > > > > > >> glibc, > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > >>> > > > > > >>> * More general solution by rebuilding the image to install > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > > >>> > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > eventually. > > > > > Thus, > > > > > >>> I begin to think whether we should change our Dockerfile to > adopt > > > > > >> jemalloc > > > > > >>> as default memory allocator [4]. > > > > > >>> From my point of view, we have two choices: > > > > > >>> > > > > > >>> 1. Introduce another Dockerfile using jemalloc as default > > > memory > > > > > >>> allocator, which means Flink needs another two new image tags > to > > > > build > > > > > >>> docker with jemalloc while default docker still use glibc. > > > > > >>> 2. Set the default memory allocator as jemalloc in our > > existing > > > > > >>> Dockerfiles, which means Flink offer docker image with jemalloc > > by > > > > > >> default. > > > > > >>> I prefer the 2nd option as our company already use jemalloc as > > > > default > > > > > >>> memory allocator for JDK at our production environment due to > > > > messages > > > > > >> from > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > >>> Moreover, I found several open source projects adopting > jemalloc > > as > > > > > >>> default memory allocator within their images to resolve memory > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > > >>> > > > > > >>> What do you guys think of this issue? > > > > > >>> > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > >>> [2] > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > >>> [5] > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > >>> > > > > > >>> > > > > > >>> Best > > > > > >>> Yun Tang > > > > > >>> > > > > > > > > > > > > > > > > > > > > |
|
|
Hi
I think Till's view deserves wider discussion. I want to give my two cents when I debug with Nico on his reported RocksDB OOM problem. Jemalloc has the mechanism to profile memory allocation [1] which is widely used to analysis memory leak. Once we set jemalloc as default memory allocator, the frequency of OOM behavior decreases obviously. Considering the OOM killed problem in k8s, change default memory allocator as jemalloc could be something beneficial. [1] https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling Best Yun Tang ________________________________ From: Till Rohrmann <[hidden email]> Sent: Thursday, October 29, 2020 18:34 To: dev <[hidden email]> Cc: Yun Tang <[hidden email]> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Hi Yu, I see your point why you are in favour of continuing using glibc. I honestly don't have a lot of experience with malloc libraries so I can only argue from a general perspective: We know that glibc has some problems wrt to memory fragmentation which can cause processes to exceed its memory limit. jemalloc seems to address this problem by avoiding memory fragmentation. However, it might be the case that jemalloc introduces other problems. In order to figure this out we would need to collect more data points/experience. So in the end we have to weigh glibc memory fragmentation against unknown problems in jemalloc. Given that choosing the malloc library will be quite an expert setting, I don't expect many Flink users to use it in case they see OOMs. Hence, I would be slightly in favour of choosing jemalloc as the default because it would allow us to gather feedback from users and solve the existing problem. In the best case, user's won't run into new problems. If they do, then we will have to re-evaluate this decision and might have to switch the malloc library back. W/o setting jemalloc as the default, I fear that we will never gather enough feedback to make jemalloc confidently the default. What one could argue, though, is that making jemalloc the default so shortly before the feature freeze limits the timespan we can observe how it behaves before the actual release. Doing such a change at the beginning of a release cycle, would allow us to gain more confidence. In any way, I believe that we should add a big release note and document what users should do in case they see memory issues (OOM kills, slower performance, etc.). Cheers, Till On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]> wrote: > True, thanks for the reminder Till! > > I suggest using glibc malloc as default. On one hand this follows our old > behavior (only with glibc malloc support in the image), on the other hand I > believe glibc isn't using jemalloc as its default memory allocator for some > reason. > > Please let me know your thoughts. Thanks. > > Best Regards, > Yu > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email]> wrote: > > > The only question left would be what will be the default value? > > > > Cheers, > > Till > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]> wrote: > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > It seems we have reached consensus and there are already enough +1s to > > move > > > forward, and suggest @Yun to conclude the discussion directly if there > > are > > > no objections. > > > > > > Thanks. > > > > > > Best Regards, > > > Yu > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> > > wrote: > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > [hidden email]> > > > > wrote: > > > > > > > > > If it is possible to support both allocators in a single image then > > we > > > > > should definitely go with that option. > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > Thanks for Yang's suggestion. I think this could be a better > > choice. > > > > > > We could install jemalloc and only enable it in LD_PRELOAD when > > user > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > By doing so, we could avoid to create another docker image tags > and > > > > also > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > Best > > > > > > Yun Tang > > > > > > ________________________________ > > > > > > From: Yang Wang <[hidden email]> > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > To: dev <[hidden email]> > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > I think this is very important when deploy Flink with container > > > > > environment > > > > > > in production. I just > > > > > > have quick question. Could we have both memory allocator(e.g. > > glibc, > > > > > > jemalloc) in the Flink > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > Best, > > > > > > Yang > > > > > > > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > > discussion > > > > > >> Yun! > > > > > >> > > > > > >> For the given solutions, I prefer option 1 (supply another > > > Dockerfile > > > > > using > > > > > >> jemalloc as default memory allocator) because of the below > > reasons: > > > > > >> > > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc > (glibc > > > > > malloc), > > > > > >> or else glibc should have already adopted it as the default > memory > > > > > >> allocator. And as indicated here [1], in some cases jemalloc > will > > > > > >> consume as much as twice the memory than glibc > > > > > >> > > > > > >> 2. All existing Flink docker images use glibc, if we change the > > > > default > > > > > >> memory allocator to jemalloc and only supply one series of > images, > > > we > > > > > will > > > > > >> leave those having better performance with glibc no other > choices > > > but > > > > > >> staying with old images. In another word, there's a risk of > > > > introducing > > > > > new > > > > > >> problems while fixing an existing one if choosing option-2. > > > > > >> > > > > > >> And there is a third option considering the efforts of > maintaining > > > > more > > > > > >> images if the memory leak issue is not widely observed, that we > > > could > > > > > >> document the steps of building Dockerfile with jemalloc as > default > > > > > >> allocator so users could build it when needed, which leaves the > > > burden > > > > > to > > > > > >> our users so for me it's not the best option. > > > > > >> > > > > > >> Best Regards, > > > > > >> Yu > > > > > >> > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > >> > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> > wrote: > > > > > >> > > > > > >>> Hi all > > > > > >>> > > > > > >>> Users report they meet serious memory leak when submitting jobs > > > > > >>> continously in session mode within k8s (please refer to > > > > FLINK-18712[1] > > > > > ), > > > > > >>> and I also reproduce this to find this is caused by memory > > > > > fragmentation > > > > > >> of > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > >>> > > > > > >>> * Quick but not very clean solution to limit the memory > pool > > > of > > > > > >> glibc, > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > >>> > > > > > >>> * More general solution by rebuilding the image to install > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > > >>> > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > eventually. > > > > > Thus, > > > > > >>> I begin to think whether we should change our Dockerfile to > adopt > > > > > >> jemalloc > > > > > >>> as default memory allocator [4]. > > > > > >>> From my point of view, we have two choices: > > > > > >>> > > > > > >>> 1. Introduce another Dockerfile using jemalloc as default > > > memory > > > > > >>> allocator, which means Flink needs another two new image tags > to > > > > build > > > > > >>> docker with jemalloc while default docker still use glibc. > > > > > >>> 2. Set the default memory allocator as jemalloc in our > > existing > > > > > >>> Dockerfiles, which means Flink offer docker image with jemalloc > > by > > > > > >> default. > > > > > >>> I prefer the 2nd option as our company already use jemalloc as > > > > default > > > > > >>> memory allocator for JDK at our production environment due to > > > > messages > > > > > >> from > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > >>> Moreover, I found several open source projects adopting > jemalloc > > as > > > > > >>> default memory allocator within their images to resolve memory > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > > >>> > > > > > >>> What do you guys think of this issue? > > > > > >>> > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > >>> [2] > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > >>> [5] > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > >>> > > > > > >>> > > > > > >>> Best > > > > > >>> Yun Tang > > > > > >>> > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
|
I think Till has a good point. The default should work well, because very
few users will ever change it. Even on OOMs. So I am slightly in favor of going with jemalloc. @Yun Tang - Do you see a noticeable performance difference between the two? On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]> wrote: > Hi > > I think Till's view deserves wider discussion. > I want to give my two cents when I debug with Nico on his reported RocksDB > OOM problem. > Jemalloc has the mechanism to profile memory allocation [1] which is > widely used to analysis memory leak. > Once we set jemalloc as default memory allocator, the frequency of OOM > behavior decreases obviously. > > Considering the OOM killed problem in k8s, change default memory allocator > as jemalloc could be something beneficial. > > [1] https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > Best > Yun Tang > > ________________________________ > From: Till Rohrmann <[hidden email]> > Sent: Thursday, October 29, 2020 18:34 > To: dev <[hidden email]> > Cc: Yun Tang <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for debian based Flink docker image > > Hi Yu, > > I see your point why you are in favour of continuing using glibc. > > I honestly don't have a lot of experience with malloc libraries so I can > only argue from a general perspective: We know that glibc has some problems > wrt to memory fragmentation which can cause processes to exceed its memory > limit. jemalloc seems to address this problem by avoiding memory > fragmentation. However, it might be the case that jemalloc introduces other > problems. In order to figure this out we would need to collect more data > points/experience. So in the end we have to weigh glibc memory > fragmentation against unknown problems in jemalloc. > > Given that choosing the malloc library will be quite an expert setting, I > don't expect many Flink users to use it in case they see OOMs. Hence, I > would be slightly in favour of choosing jemalloc as the default because it > would allow us to gather feedback from users and solve the existing > problem. In the best case, user's won't run into new problems. If they do, > then we will have to re-evaluate this decision and might have to switch the > malloc library back. W/o setting jemalloc as the default, I fear that we > will never gather enough feedback to make jemalloc confidently the default. > > What one could argue, though, is that making jemalloc the default so > shortly before the feature freeze limits the timespan we can observe how it > behaves before the actual release. Doing such a change at the beginning of > a release cycle, would allow us to gain more confidence. > > In any way, I believe that we should add a big release note and document > what users should do in case they see memory issues (OOM kills, slower > performance, etc.). > > Cheers, > Till > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]> wrote: > > > True, thanks for the reminder Till! > > > > I suggest using glibc malloc as default. On one hand this follows our old > > behavior (only with glibc malloc support in the image), on the other > hand I > > believe glibc isn't using jemalloc as its default memory allocator for > some > > reason. > > > > Please let me know your thoughts. Thanks. > > > > Best Regards, > > Yu > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email]> > wrote: > > > > > The only question left would be what will be the default value? > > > > > > Cheers, > > > Till > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]> wrote: > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > It seems we have reached consensus and there are already enough +1s > to > > > move > > > > forward, and suggest @Yun to conclude the discussion directly if > there > > > are > > > > no objections. > > > > > > > > Thanks. > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > [hidden email]> > > > > > wrote: > > > > > > > > > > > If it is possible to support both allocators in a single image > then > > > we > > > > > > should definitely go with that option. > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > Thanks for Yang's suggestion. I think this could be a better > > > choice. > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD when > > > user > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > By doing so, we could avoid to create another docker image tags > > and > > > > > also > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > Best > > > > > > > Yun Tang > > > > > > > ________________________________ > > > > > > > From: Yang Wang <[hidden email]> > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > To: dev <[hidden email]> > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > I think this is very important when deploy Flink with container > > > > > > environment > > > > > > > in production. I just > > > > > > > have quick question. Could we have both memory allocator(e.g. > > > glibc, > > > > > > > jemalloc) in the Flink > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > Best, > > > > > > > Yang > > > > > > > > > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > > > discussion > > > > > > >> Yun! > > > > > > >> > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > Dockerfile > > > > > > using > > > > > > >> jemalloc as default memory allocator) because of the below > > > reasons: > > > > > > >> > > > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc > > (glibc > > > > > > malloc), > > > > > > >> or else glibc should have already adopted it as the default > > memory > > > > > > >> allocator. And as indicated here [1], in some cases jemalloc > > will > > > > > > >> consume as much as twice the memory than glibc > > > > > > >> > > > > > > >> 2. All existing Flink docker images use glibc, if we change > the > > > > > default > > > > > > >> memory allocator to jemalloc and only supply one series of > > images, > > > > we > > > > > > will > > > > > > >> leave those having better performance with glibc no other > > choices > > > > but > > > > > > >> staying with old images. In another word, there's a risk of > > > > > introducing > > > > > > new > > > > > > >> problems while fixing an existing one if choosing option-2. > > > > > > >> > > > > > > >> And there is a third option considering the efforts of > > maintaining > > > > > more > > > > > > >> images if the memory leak issue is not widely observed, that > we > > > > could > > > > > > >> document the steps of building Dockerfile with jemalloc as > > default > > > > > > >> allocator so users could build it when needed, which leaves > the > > > > burden > > > > > > to > > > > > > >> our users so for me it's not the best option. > > > > > > >> > > > > > > >> Best Regards, > > > > > > >> Yu > > > > > > >> > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > >> > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> > > wrote: > > > > > > >> > > > > > > >>> Hi all > > > > > > >>> > > > > > > >>> Users report they meet serious memory leak when submitting > jobs > > > > > > >>> continously in session mode within k8s (please refer to > > > > > FLINK-18712[1] > > > > > > ), > > > > > > >>> and I also reproduce this to find this is caused by memory > > > > > > fragmentation > > > > > > >> of > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > >>> > > > > > > >>> * Quick but not very clean solution to limit the memory > > pool > > > > of > > > > > > >> glibc, > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > >>> > > > > > > >>> * More general solution by rebuilding the image to > install > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > > > >>> > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > eventually. > > > > > > Thus, > > > > > > >>> I begin to think whether we should change our Dockerfile to > > adopt > > > > > > >> jemalloc > > > > > > >>> as default memory allocator [4]. > > > > > > >>> From my point of view, we have two choices: > > > > > > >>> > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as default > > > > memory > > > > > > >>> allocator, which means Flink needs another two new image tags > > to > > > > > build > > > > > > >>> docker with jemalloc while default docker still use glibc. > > > > > > >>> 2. Set the default memory allocator as jemalloc in our > > > existing > > > > > > >>> Dockerfiles, which means Flink offer docker image with > jemalloc > > > by > > > > > > >> default. > > > > > > >>> I prefer the 2nd option as our company already use jemalloc > as > > > > > default > > > > > > >>> memory allocator for JDK at our production environment due to > > > > > messages > > > > > > >> from > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > >>> Moreover, I found several open source projects adopting > > jemalloc > > > as > > > > > > >>> default memory allocator within their images to resolve > memory > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > > > >>> > > > > > > >>> What do you guys think of this issue? > > > > > > >>> > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > >>> [2] > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > >>> [5] > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > >>> > > > > > > >>> > > > > > > >>> Best > > > > > > >>> Yun Tang > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
Thanks for sharing more thoughts guys.
My experience with malloc libraries are also limited, and as mentioned in my previous email [1], my major concern comes from some online information [2] which indicates in some cases jemalloc will consume as much as twice the memory than glibc, which is a huge cost from my point of view. However, one may also argue that this post was written in 2015 and things might have been improved a lot ever since. Nevertheless, as Till and Yun pointed out, I could also see the benefits of setting jemalloc as default, so I don't have strong objections here. But this actually is a change of the default memory allocator of our docker image (previously glibc is the only choice) which definitely worth a big release note. Best Regards, Yu [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html [2] https://stackoverflow.com/a/33993215 On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]> wrote: > I think Till has a good point. The default should work well, because very > few users will ever change it. Even on OOMs. > So I am slightly in favor of going with jemalloc. > > @Yun Tang - Do you see a noticeable performance difference between the two? > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]> wrote: > > > Hi > > > > I think Till's view deserves wider discussion. > > I want to give my two cents when I debug with Nico on his reported > RocksDB > > OOM problem. > > Jemalloc has the mechanism to profile memory allocation [1] which is > > widely used to analysis memory leak. > > Once we set jemalloc as default memory allocator, the frequency of OOM > > behavior decreases obviously. > > > > Considering the OOM killed problem in k8s, change default memory > allocator > > as jemalloc could be something beneficial. > > > > [1] https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > Best > > Yun Tang > > > > ________________________________ > > From: Till Rohrmann <[hidden email]> > > Sent: Thursday, October 29, 2020 18:34 > > To: dev <[hidden email]> > > Cc: Yun Tang <[hidden email]> > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > > for debian based Flink docker image > > > > Hi Yu, > > > > I see your point why you are in favour of continuing using glibc. > > > > I honestly don't have a lot of experience with malloc libraries so I can > > only argue from a general perspective: We know that glibc has some > problems > > wrt to memory fragmentation which can cause processes to exceed its > memory > > limit. jemalloc seems to address this problem by avoiding memory > > fragmentation. However, it might be the case that jemalloc introduces > other > > problems. In order to figure this out we would need to collect more data > > points/experience. So in the end we have to weigh glibc memory > > fragmentation against unknown problems in jemalloc. > > > > Given that choosing the malloc library will be quite an expert setting, I > > don't expect many Flink users to use it in case they see OOMs. Hence, I > > would be slightly in favour of choosing jemalloc as the default because > it > > would allow us to gather feedback from users and solve the existing > > problem. In the best case, user's won't run into new problems. If they > do, > > then we will have to re-evaluate this decision and might have to switch > the > > malloc library back. W/o setting jemalloc as the default, I fear that we > > will never gather enough feedback to make jemalloc confidently the > default. > > > > What one could argue, though, is that making jemalloc the default so > > shortly before the feature freeze limits the timespan we can observe how > it > > behaves before the actual release. Doing such a change at the beginning > of > > a release cycle, would allow us to gain more confidence. > > > > In any way, I believe that we should add a big release note and document > > what users should do in case they see memory issues (OOM kills, slower > > performance, etc.). > > > > Cheers, > > Till > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]> wrote: > > > > > True, thanks for the reminder Till! > > > > > > I suggest using glibc malloc as default. On one hand this follows our > old > > > behavior (only with glibc malloc support in the image), on the other > > hand I > > > believe glibc isn't using jemalloc as its default memory allocator for > > some > > > reason. > > > > > > Please let me know your thoughts. Thanks. > > > > > > Best Regards, > > > Yu > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email]> > > wrote: > > > > > > > The only question left would be what will be the default value? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]> wrote: > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > It seems we have reached consensus and there are already enough +1s > > to > > > > move > > > > > forward, and suggest @Yun to conclude the discussion directly if > > there > > > > are > > > > > no objections. > > > > > > > > > > Thanks. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > If it is possible to support both allocators in a single image > > then > > > > we > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > Thanks for Yang's suggestion. I think this could be a better > > > > choice. > > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD > when > > > > user > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > By doing so, we could avoid to create another docker image > tags > > > and > > > > > > also > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > Best > > > > > > > > Yun Tang > > > > > > > > ________________________________ > > > > > > > > From: Yang Wang <[hidden email]> > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > To: dev <[hidden email]> > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > memory > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > container > > > > > > > environment > > > > > > > > in production. I just > > > > > > > > have quick question. Could we have both memory allocator(e.g. > > > > glibc, > > > > > > > > jemalloc) in the Flink > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > Best, > > > > > > > > Yang > > > > > > > > > > > > > > > > Yu Li <[hidden email]> 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > > > > discussion > > > > > > > >> Yun! > > > > > > > >> > > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > > Dockerfile > > > > > > > using > > > > > > > >> jemalloc as default memory allocator) because of the below > > > > reasons: > > > > > > > >> > > > > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc > > > (glibc > > > > > > > malloc), > > > > > > > >> or else glibc should have already adopted it as the default > > > memory > > > > > > > >> allocator. And as indicated here [1], in some cases jemalloc > > > will > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > >> > > > > > > > >> 2. All existing Flink docker images use glibc, if we change > > the > > > > > > default > > > > > > > >> memory allocator to jemalloc and only supply one series of > > > images, > > > > > we > > > > > > > will > > > > > > > >> leave those having better performance with glibc no other > > > choices > > > > > but > > > > > > > >> staying with old images. In another word, there's a risk of > > > > > > introducing > > > > > > > new > > > > > > > >> problems while fixing an existing one if choosing option-2. > > > > > > > >> > > > > > > > >> And there is a third option considering the efforts of > > > maintaining > > > > > > more > > > > > > > >> images if the memory leak issue is not widely observed, that > > we > > > > > could > > > > > > > >> document the steps of building Dockerfile with jemalloc as > > > default > > > > > > > >> allocator so users could build it when needed, which leaves > > the > > > > > burden > > > > > > > to > > > > > > > >> our users so for me it's not the best option. > > > > > > > >> > > > > > > > >> Best Regards, > > > > > > > >> Yu > > > > > > > >> > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > >> > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]> > > > wrote: > > > > > > > >> > > > > > > > >>> Hi all > > > > > > > >>> > > > > > > > >>> Users report they meet serious memory leak when submitting > > jobs > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > FLINK-18712[1] > > > > > > > ), > > > > > > > >>> and I also reproduce this to find this is caused by memory > > > > > > > fragmentation > > > > > > > >> of > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > >>> > > > > > > > >>> * Quick but not very clean solution to limit the > memory > > > pool > > > > > of > > > > > > > >> glibc, > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > >>> > > > > > > > >>> * More general solution by rebuilding the image to > > install > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > > > > >>> > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > eventually. > > > > > > > Thus, > > > > > > > >>> I begin to think whether we should change our Dockerfile to > > > adopt > > > > > > > >> jemalloc > > > > > > > >>> as default memory allocator [4]. > > > > > > > >>> From my point of view, we have two choices: > > > > > > > >>> > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > default > > > > > memory > > > > > > > >>> allocator, which means Flink needs another two new image > tags > > > to > > > > > > build > > > > > > > >>> docker with jemalloc while default docker still use glibc. > > > > > > > >>> 2. Set the default memory allocator as jemalloc in our > > > > existing > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > jemalloc > > > > by > > > > > > > >> default. > > > > > > > >>> I prefer the 2nd option as our company already use jemalloc > > as > > > > > > default > > > > > > > >>> memory allocator for JDK at our production environment due > to > > > > > > messages > > > > > > > >> from > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > >>> Moreover, I found several open source projects adopting > > > jemalloc > > > > as > > > > > > > >>> default memory allocator within their images to resolve > > memory > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > > > > >>> > > > > > > > >>> What do you guys think of this issue? > > > > > > > >>> > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > >>> [2] > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > >>> [5] > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Best > > > > > > > >>> Yun Tang > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
we also met glic memory leak .it appeared many 64 MB memory segment when using pmap command . i am +1 on setting jemalloc as default
发送自autohome ________________________________ 发件人: Yu Li <[hidden email]> 发送时间: 2020-10-30 00:15:12 收件人: dev <[hidden email]> 抄送: [hidden email] <[hidden email]> 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Thanks for sharing more thoughts guys. My experience with malloc libraries are also limited, and as mentioned in my previous email [1], my major concern comes from some online information [2] which indicates in some cases jemalloc will consume as much as twice the memory than glibc, which is a huge cost from my point of view. However, one may also argue that this post was written in 2015 and things might have been improved a lot ever since. Nevertheless, as Till and Yun pointed out, I could also see the benefits of setting jemalloc as default, so I don't have strong objections here. But this actually is a change of the default memory allocator of our docker image (previously glibc is the only choice) which definitely worth a big release note. Best Regards, Yu [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html [2] https://stackoverflow.com/a/33993215 On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto:[hidden email]>> wrote: > I think Till has a good point. The default should work well, because very > few users will ever change it. Even on OOMs. > So I am slightly in favor of going with jemalloc. > > @Yun Tang - Do you see a noticeable performance difference between the two? > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto:[hidden email]>> wrote: > > > Hi > > > > I think Till's view deserves wider discussion. > > I want to give my two cents when I debug with Nico on his reported > RocksDB > > OOM problem. > > Jemalloc has the mechanism to profile memory allocation [1] which is > > widely used to analysis memory leak. > > Once we set jemalloc as default memory allocator, the frequency of OOM > > behavior decreases obviously. > > > > Considering the OOM killed problem in k8s, change default memory > allocator > > as jemalloc could be something beneficial. > > > > [1] https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > Best > > Yun Tang > > > > ________________________________ > > From: Till Rohrmann <[hidden email]<mailto:[hidden email]>> > > Sent: Thursday, October 29, 2020 18:34 > > To: dev <[hidden email]<mailto:[hidden email]>> > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > > for debian based Flink docker image > > > > Hi Yu, > > > > I see your point why you are in favour of continuing using glibc. > > > > I honestly don't have a lot of experience with malloc libraries so I can > > only argue from a general perspective: We know that glibc has some > problems > > wrt to memory fragmentation which can cause processes to exceed its > memory > > limit. jemalloc seems to address this problem by avoiding memory > > fragmentation. However, it might be the case that jemalloc introduces > other > > problems. In order to figure this out we would need to collect more data > > points/experience. So in the end we have to weigh glibc memory > > fragmentation against unknown problems in jemalloc. > > > > Given that choosing the malloc library will be quite an expert setting, I > > don't expect many Flink users to use it in case they see OOMs. Hence, I > > would be slightly in favour of choosing jemalloc as the default because > it > > would allow us to gather feedback from users and solve the existing > > problem. In the best case, user's won't run into new problems. If they > do, > > then we will have to re-evaluate this decision and might have to switch > the > > malloc library back. W/o setting jemalloc as the default, I fear that we > > will never gather enough feedback to make jemalloc confidently the > default. > > > > What one could argue, though, is that making jemalloc the default so > > shortly before the feature freeze limits the timespan we can observe how > it > > behaves before the actual release. Doing such a change at the beginning > of > > a release cycle, would allow us to gain more confidence. > > > > In any way, I believe that we should add a big release note and document > > what users should do in case they see memory issues (OOM kills, slower > > performance, etc.). > > > > Cheers, > > Till > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto:[hidden email]>> wrote: > > > > > True, thanks for the reminder Till! > > > > > > I suggest using glibc malloc as default. On one hand this follows our > old > > > behavior (only with glibc malloc support in the image), on the other > > hand I > > > believe glibc isn't using jemalloc as its default memory allocator for > > some > > > reason. > > > > > > Please let me know your thoughts. Thanks. > > > > > > Best Regards, > > > Yu > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email]<mailto:[hidden email]>> > > wrote: > > > > > > > The only question left would be what will be the default value? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]<mailto:[hidden email]>> wrote: > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > It seems we have reached consensus and there are already enough +1s > > to > > > > move > > > > > forward, and suggest @Yun to conclude the discussion directly if > > there > > > > are > > > > > no objections. > > > > > > > > > > Thanks. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <[hidden email]<mailto:[hidden email]>> > > > > wrote: > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > wrote: > > > > > > > > > > > > > If it is possible to support both allocators in a single image > > then > > > > we > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > Thanks for Yang's suggestion. I think this could be a better > > > > choice. > > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD > when > > > > user > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > By doing so, we could avoid to create another docker image > tags > > > and > > > > > > also > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > Best > > > > > > > > Yun Tang > > > > > > > > ________________________________ > > > > > > > > From: Yang Wang <[hidden email]<mailto:[hidden email]>> > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > memory > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > container > > > > > > > environment > > > > > > > > in production. I just > > > > > > > > have quick question. Could we have both memory allocator(e.g. > > > > glibc, > > > > > > > > jemalloc) in the Flink > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > Best, > > > > > > > > Yang > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving the > > > > > > discussion > > > > > > > >> Yun! > > > > > > > >> > > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > > Dockerfile > > > > > > > using > > > > > > > >> jemalloc as default memory allocator) because of the below > > > > reasons: > > > > > > > >> > > > > > > > >> 1. It's hard to say jemalloc is always better than ptmalloc > > > (glibc > > > > > > > malloc), > > > > > > > >> or else glibc should have already adopted it as the default > > > memory > > > > > > > >> allocator. And as indicated here [1], in some cases jemalloc > > > will > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > >> > > > > > > > >> 2. All existing Flink docker images use glibc, if we change > > the > > > > > > default > > > > > > > >> memory allocator to jemalloc and only supply one series of > > > images, > > > > > we > > > > > > > will > > > > > > > >> leave those having better performance with glibc no other > > > choices > > > > > but > > > > > > > >> staying with old images. In another word, there's a risk of > > > > > > introducing > > > > > > > new > > > > > > > >> problems while fixing an existing one if choosing option-2. > > > > > > > >> > > > > > > > >> And there is a third option considering the efforts of > > > maintaining > > > > > > more > > > > > > > >> images if the memory leak issue is not widely observed, that > > we > > > > > could > > > > > > > >> document the steps of building Dockerfile with jemalloc as > > > default > > > > > > > >> allocator so users could build it when needed, which leaves > > the > > > > > burden > > > > > > > to > > > > > > > >> our users so for me it's not the best option. > > > > > > > >> > > > > > > > >> Best Regards, > > > > > > > >> Yu > > > > > > > >> > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > >> > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email]<mailto:[hidden email]>> > > > wrote: > > > > > > > >> > > > > > > > >>> Hi all > > > > > > > >>> > > > > > > > >>> Users report they meet serious memory leak when submitting > > jobs > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > FLINK-18712[1] > > > > > > > ), > > > > > > > >>> and I also reproduce this to find this is caused by memory > > > > > > > fragmentation > > > > > > > >> of > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > >>> > > > > > > > >>> * Quick but not very clean solution to limit the > memory > > > pool > > > > > of > > > > > > > >> glibc, > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > >>> > > > > > > > >>> * More general solution by rebuilding the image to > > install > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to LD_PRELOAD > > > > > > > >>> > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > eventually. > > > > > > > Thus, > > > > > > > >>> I begin to think whether we should change our Dockerfile to > > > adopt > > > > > > > >> jemalloc > > > > > > > >>> as default memory allocator [4]. > > > > > > > >>> From my point of view, we have two choices: > > > > > > > >>> > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > default > > > > > memory > > > > > > > >>> allocator, which means Flink needs another two new image > tags > > > to > > > > > > build > > > > > > > >>> docker with jemalloc while default docker still use glibc. > > > > > > > >>> 2. Set the default memory allocator as jemalloc in our > > > > existing > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > jemalloc > > > > by > > > > > > > >> default. > > > > > > > >>> I prefer the 2nd option as our company already use jemalloc > > as > > > > > > default > > > > > > > >>> memory allocator for JDK at our production environment due > to > > > > > > messages > > > > > > > >> from > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > >>> Moreover, I found several open source projects adopting > > > jemalloc > > > > as > > > > > > > >>> default memory allocator within their images to resolve > > memory > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant [6]. > > > > > > > >>> > > > > > > > >>> What do you guys think of this issue? > > > > > > > >>> > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > >>> [2] > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > >>> [3] https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > >>> [5] > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Best > > > > > > > >>> Yun Tang > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
After long time running test application(>96 hours) , glibc used less
memory than jemalloc , almost half of jemalloc. Can refer this test case. Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]> wrote: > we also met glic memory leak .it appeared many 64 MB memory segment when > using pmap command . i am +1 on setting jemalloc as default > > 发送自autohome > ________________________________ > 发件人: Yu Li <[hidden email]> > 发送时间: 2020-10-30 00:15:12 > 收件人: dev <[hidden email]> > 抄送: [hidden email] <[hidden email]> > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for > debian based Flink docker image > > Thanks for sharing more thoughts guys. > > My experience with malloc libraries are also limited, and as mentioned in > my previous email [1], my major concern comes from some online information > [2] which indicates in some cases jemalloc will consume as much as twice > the memory than glibc, which is a huge cost from my point of view. However, > one may also argue that this post was written in 2015 and things might have > been improved a lot ever since. > > Nevertheless, as Till and Yun pointed out, I could also see the benefits of > setting jemalloc as default, so I don't have strong objections here. But > this actually is a change of the default memory allocator of our docker > image (previously glibc is the only choice) which definitely worth a big > release note. > > Best Regards, > Yu > > [1] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > [2] https://stackoverflow.com/a/33993215 > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto: > [hidden email]>> wrote: > > > I think Till has a good point. The default should work well, because very > > few users will ever change it. Even on OOMs. > > So I am slightly in favor of going with jemalloc. > > > > @Yun Tang - Do you see a noticeable performance difference between the > two? > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto: > [hidden email]>> wrote: > > > > > Hi > > > > > > I think Till's view deserves wider discussion. > > > I want to give my two cents when I debug with Nico on his reported > > RocksDB > > > OOM problem. > > > Jemalloc has the mechanism to profile memory allocation [1] which is > > > widely used to analysis memory leak. > > > Once we set jemalloc as default memory allocator, the frequency of OOM > > > behavior decreases obviously. > > > > > > Considering the OOM killed problem in k8s, change default memory > > allocator > > > as jemalloc could be something beneficial. > > > > > > [1] > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > Best > > > Yun Tang > > > > > > ________________________________ > > > From: Till Rohrmann <[hidden email]<mailto:[hidden email] > >> > > > Sent: Thursday, October 29, 2020 18:34 > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > allocator > > > for debian based Flink docker image > > > > > > Hi Yu, > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > I honestly don't have a lot of experience with malloc libraries so I > can > > > only argue from a general perspective: We know that glibc has some > > problems > > > wrt to memory fragmentation which can cause processes to exceed its > > memory > > > limit. jemalloc seems to address this problem by avoiding memory > > > fragmentation. However, it might be the case that jemalloc introduces > > other > > > problems. In order to figure this out we would need to collect more > data > > > points/experience. So in the end we have to weigh glibc memory > > > fragmentation against unknown problems in jemalloc. > > > > > > Given that choosing the malloc library will be quite an expert > setting, I > > > don't expect many Flink users to use it in case they see OOMs. Hence, I > > > would be slightly in favour of choosing jemalloc as the default because > > it > > > would allow us to gather feedback from users and solve the existing > > > problem. In the best case, user's won't run into new problems. If they > > do, > > > then we will have to re-evaluate this decision and might have to switch > > the > > > malloc library back. W/o setting jemalloc as the default, I fear that > we > > > will never gather enough feedback to make jemalloc confidently the > > default. > > > > > > What one could argue, though, is that making jemalloc the default so > > > shortly before the feature freeze limits the timespan we can observe > how > > it > > > behaves before the actual release. Doing such a change at the beginning > > of > > > a release cycle, would allow us to gain more confidence. > > > > > > In any way, I believe that we should add a big release note and > document > > > what users should do in case they see memory issues (OOM kills, slower > > > performance, etc.). > > > > > > Cheers, > > > Till > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto: > [hidden email]>> wrote: > > > > > > > True, thanks for the reminder Till! > > > > > > > > I suggest using glibc malloc as default. On one hand this follows our > > old > > > > behavior (only with glibc malloc support in the image), on the other > > > hand I > > > > believe glibc isn't using jemalloc as its default memory allocator > for > > > some > > > > reason. > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email] > <mailto:[hidden email]>> > > > wrote: > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]<mailto: > [hidden email]>> wrote: > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > > > It seems we have reached consensus and there are already enough > +1s > > > to > > > > > move > > > > > > forward, and suggest @Yun to conclude the discussion directly if > > > there > > > > > are > > > > > > no objections. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Best Regards, > > > > > > Yu > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > [hidden email]<mailto:[hidden email]>> > > > > > wrote: > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > > wrote: > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > image > > > then > > > > > we > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > better > > > > > choice. > > > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD > > when > > > > > user > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > By doing so, we could avoid to create another docker image > > tags > > > > and > > > > > > > also > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > Best > > > > > > > > > Yun Tang > > > > > > > > > ________________________________ > > > > > > > > > From: Yang Wang <[hidden email]<mailto: > [hidden email]>> > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > To: dev <[hidden email]<mailto:[hidden email] > >> > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > > memory > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > container > > > > > > > > environment > > > > > > > > > in production. I just > > > > > > > > > have quick question. Could we have both memory > allocator(e.g. > > > > > glibc, > > > > > > > > > jemalloc) in the Flink > > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving > the > > > > > > > discussion > > > > > > > > >> Yun! > > > > > > > > >> > > > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > > > Dockerfile > > > > > > > > using > > > > > > > > >> jemalloc as default memory allocator) because of the below > > > > > reasons: > > > > > > > > >> > > > > > > > > >> 1. It's hard to say jemalloc is always better than > ptmalloc > > > > (glibc > > > > > > > > malloc), > > > > > > > > >> or else glibc should have already adopted it as the > default > > > > memory > > > > > > > > >> allocator. And as indicated here [1], in some cases > jemalloc > > > > will > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > >> > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > change > > > the > > > > > > > default > > > > > > > > >> memory allocator to jemalloc and only supply one series of > > > > images, > > > > > > we > > > > > > > > will > > > > > > > > >> leave those having better performance with glibc no other > > > > choices > > > > > > but > > > > > > > > >> staying with old images. In another word, there's a risk > of > > > > > > > introducing > > > > > > > > new > > > > > > > > >> problems while fixing an existing one if choosing > option-2. > > > > > > > > >> > > > > > > > > >> And there is a third option considering the efforts of > > > > maintaining > > > > > > > more > > > > > > > > >> images if the memory leak issue is not widely observed, > that > > > we > > > > > > could > > > > > > > > >> document the steps of building Dockerfile with jemalloc as > > > > default > > > > > > > > >> allocator so users could build it when needed, which > leaves > > > the > > > > > > burden > > > > > > > > to > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > >> > > > > > > > > >> Best Regards, > > > > > > > > >> Yu > > > > > > > > >> > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > >> > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email] > <mailto:[hidden email]>> > > > > wrote: > > > > > > > > >> > > > > > > > > >>> Hi all > > > > > > > > >>> > > > > > > > > >>> Users report they meet serious memory leak when > submitting > > > jobs > > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > > FLINK-18712[1] > > > > > > > > ), > > > > > > > > >>> and I also reproduce this to find this is caused by > memory > > > > > > > > fragmentation > > > > > > > > >> of > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > >>> > > > > > > > > >>> * Quick but not very clean solution to limit the > > memory > > > > pool > > > > > > of > > > > > > > > >> glibc, > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > >>> > > > > > > > > >>> * More general solution by rebuilding the image to > > > install > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > LD_PRELOAD > > > > > > > > >>> > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > > eventually. > > > > > > > > Thus, > > > > > > > > >>> I begin to think whether we should change our Dockerfile > to > > > > adopt > > > > > > > > >> jemalloc > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > >>> > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > default > > > > > > memory > > > > > > > > >>> allocator, which means Flink needs another two new image > > tags > > > > to > > > > > > > build > > > > > > > > >>> docker with jemalloc while default docker still use > glibc. > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in > our > > > > > existing > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > > jemalloc > > > > > by > > > > > > > > >> default. > > > > > > > > >>> I prefer the 2nd option as our company already use > jemalloc > > > as > > > > > > > default > > > > > > > > >>> memory allocator for JDK at our production environment > due > > to > > > > > > > messages > > > > > > > > >> from > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > >>> Moreover, I found several open source projects adopting > > > > jemalloc > > > > > as > > > > > > > > >>> default memory allocator within their images to resolve > > > memory > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > [6]. > > > > > > > > >>> > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > >>> > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > >>> [2] > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > >>> [3] > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > >>> [5] > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> Best > > > > > > > > >>> Yun Tang > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
Hi
> Do you see a noticeable performance difference between the two? @ Stephan Ewen , as we already use jemalloc as default memory allocator in production, we do not have much experience to compare performace between glibc and jemalloc. And I did not take a look at the performance difference when I debug docker OOM. I'll have a try to run benchmark on docker with different allocators when I have time these days. @wang gang, yes, that is what I also observed when I pmap the memory when using glibc, many memory segments with 64MB size. @ Billy Xie, what kind of test case did you use? In my point of view, compared to who would use more memory in some cases, we should care more about who would behave under the max limit for most cases. Best Yun Tang ________________________________ From: Xie Billy <[hidden email]> Sent: Friday, October 30, 2020 8:46 To: [hidden email] <[hidden email]> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image After long time running test application(>96 hours) , glibc used less memory than jemalloc , almost half of jemalloc. Can refer this test case. Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]> wrote: > we also met glic memory leak .it appeared many 64 MB memory segment when > using pmap command . i am +1 on setting jemalloc as default > > 发送自autohome > ________________________________ > 发件人: Yu Li <[hidden email]> > 发送时间: 2020-10-30 00:15:12 > 收件人: dev <[hidden email]> > 抄送: [hidden email] <[hidden email]> > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for > debian based Flink docker image > > Thanks for sharing more thoughts guys. > > My experience with malloc libraries are also limited, and as mentioned in > my previous email [1], my major concern comes from some online information > [2] which indicates in some cases jemalloc will consume as much as twice > the memory than glibc, which is a huge cost from my point of view. However, > one may also argue that this post was written in 2015 and things might have > been improved a lot ever since. > > Nevertheless, as Till and Yun pointed out, I could also see the benefits of > setting jemalloc as default, so I don't have strong objections here. But > this actually is a change of the default memory allocator of our docker > image (previously glibc is the only choice) which definitely worth a big > release note. > > Best Regards, > Yu > > [1] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > [2] https://stackoverflow.com/a/33993215 > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto: > [hidden email]>> wrote: > > > I think Till has a good point. The default should work well, because very > > few users will ever change it. Even on OOMs. > > So I am slightly in favor of going with jemalloc. > > > > @Yun Tang - Do you see a noticeable performance difference between the > two? > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto: > [hidden email]>> wrote: > > > > > Hi > > > > > > I think Till's view deserves wider discussion. > > > I want to give my two cents when I debug with Nico on his reported > > RocksDB > > > OOM problem. > > > Jemalloc has the mechanism to profile memory allocation [1] which is > > > widely used to analysis memory leak. > > > Once we set jemalloc as default memory allocator, the frequency of OOM > > > behavior decreases obviously. > > > > > > Considering the OOM killed problem in k8s, change default memory > > allocator > > > as jemalloc could be something beneficial. > > > > > > [1] > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > Best > > > Yun Tang > > > > > > ________________________________ > > > From: Till Rohrmann <[hidden email]<mailto:[hidden email] > >> > > > Sent: Thursday, October 29, 2020 18:34 > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > allocator > > > for debian based Flink docker image > > > > > > Hi Yu, > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > I honestly don't have a lot of experience with malloc libraries so I > can > > > only argue from a general perspective: We know that glibc has some > > problems > > > wrt to memory fragmentation which can cause processes to exceed its > > memory > > > limit. jemalloc seems to address this problem by avoiding memory > > > fragmentation. However, it might be the case that jemalloc introduces > > other > > > problems. In order to figure this out we would need to collect more > data > > > points/experience. So in the end we have to weigh glibc memory > > > fragmentation against unknown problems in jemalloc. > > > > > > Given that choosing the malloc library will be quite an expert > setting, I > > > don't expect many Flink users to use it in case they see OOMs. Hence, I > > > would be slightly in favour of choosing jemalloc as the default because > > it > > > would allow us to gather feedback from users and solve the existing > > > problem. In the best case, user's won't run into new problems. If they > > do, > > > then we will have to re-evaluate this decision and might have to switch > > the > > > malloc library back. W/o setting jemalloc as the default, I fear that > we > > > will never gather enough feedback to make jemalloc confidently the > > default. > > > > > > What one could argue, though, is that making jemalloc the default so > > > shortly before the feature freeze limits the timespan we can observe > how > > it > > > behaves before the actual release. Doing such a change at the beginning > > of > > > a release cycle, would allow us to gain more confidence. > > > > > > In any way, I believe that we should add a big release note and > document > > > what users should do in case they see memory issues (OOM kills, slower > > > performance, etc.). > > > > > > Cheers, > > > Till > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto: > [hidden email]>> wrote: > > > > > > > True, thanks for the reminder Till! > > > > > > > > I suggest using glibc malloc as default. On one hand this follows our > > old > > > > behavior (only with glibc malloc support in the image), on the other > > > hand I > > > > believe glibc isn't using jemalloc as its default memory allocator > for > > > some > > > > reason. > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > Best Regards, > > > > Yu > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email] > <mailto:[hidden email]>> > > > wrote: > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]<mailto: > [hidden email]>> wrote: > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > > > It seems we have reached consensus and there are already enough > +1s > > > to > > > > > move > > > > > > forward, and suggest @Yun to conclude the discussion directly if > > > there > > > > > are > > > > > > no objections. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Best Regards, > > > > > > Yu > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > [hidden email]<mailto:[hidden email]>> > > > > > wrote: > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > > wrote: > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > image > > > then > > > > > we > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > better > > > > > choice. > > > > > > > > > We could install jemalloc and only enable it in LD_PRELOAD > > when > > > > > user > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > By doing so, we could avoid to create another docker image > > tags > > > > and > > > > > > > also > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > Best > > > > > > > > > Yun Tang > > > > > > > > > ________________________________ > > > > > > > > > From: Yang Wang <[hidden email]<mailto: > [hidden email]>> > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > To: dev <[hidden email]<mailto:[hidden email] > >> > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > > memory > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > container > > > > > > > > environment > > > > > > > > > in production. I just > > > > > > > > > have quick question. Could we have both memory > allocator(e.g. > > > > > glibc, > > > > > > > > > jemalloc) in the Flink > > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving > the > > > > > > > discussion > > > > > > > > >> Yun! > > > > > > > > >> > > > > > > > > >> For the given solutions, I prefer option 1 (supply another > > > > > > Dockerfile > > > > > > > > using > > > > > > > > >> jemalloc as default memory allocator) because of the below > > > > > reasons: > > > > > > > > >> > > > > > > > > >> 1. It's hard to say jemalloc is always better than > ptmalloc > > > > (glibc > > > > > > > > malloc), > > > > > > > > >> or else glibc should have already adopted it as the > default > > > > memory > > > > > > > > >> allocator. And as indicated here [1], in some cases > jemalloc > > > > will > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > >> > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > change > > > the > > > > > > > default > > > > > > > > >> memory allocator to jemalloc and only supply one series of > > > > images, > > > > > > we > > > > > > > > will > > > > > > > > >> leave those having better performance with glibc no other > > > > choices > > > > > > but > > > > > > > > >> staying with old images. In another word, there's a risk > of > > > > > > > introducing > > > > > > > > new > > > > > > > > >> problems while fixing an existing one if choosing > option-2. > > > > > > > > >> > > > > > > > > >> And there is a third option considering the efforts of > > > > maintaining > > > > > > > more > > > > > > > > >> images if the memory leak issue is not widely observed, > that > > > we > > > > > > could > > > > > > > > >> document the steps of building Dockerfile with jemalloc as > > > > default > > > > > > > > >> allocator so users could build it when needed, which > leaves > > > the > > > > > > burden > > > > > > > > to > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > >> > > > > > > > > >> Best Regards, > > > > > > > > >> Yu > > > > > > > > >> > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > >> > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <[hidden email] > <mailto:[hidden email]>> > > > > wrote: > > > > > > > > >> > > > > > > > > >>> Hi all > > > > > > > > >>> > > > > > > > > >>> Users report they meet serious memory leak when > submitting > > > jobs > > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > > FLINK-18712[1] > > > > > > > > ), > > > > > > > > >>> and I also reproduce this to find this is caused by > memory > > > > > > > > fragmentation > > > > > > > > >> of > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > >>> > > > > > > > > >>> * Quick but not very clean solution to limit the > > memory > > > > pool > > > > > > of > > > > > > > > >> glibc, > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > >>> > > > > > > > > >>> * More general solution by rebuilding the image to > > > install > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > LD_PRELOAD > > > > > > > > >>> > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > > eventually. > > > > > > > > Thus, > > > > > > > > >>> I begin to think whether we should change our Dockerfile > to > > > > adopt > > > > > > > > >> jemalloc > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > >>> > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > default > > > > > > memory > > > > > > > > >>> allocator, which means Flink needs another two new image > > tags > > > > to > > > > > > > build > > > > > > > > >>> docker with jemalloc while default docker still use > glibc. > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in > our > > > > > existing > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > > jemalloc > > > > > by > > > > > > > > >> default. > > > > > > > > >>> I prefer the 2nd option as our company already use > jemalloc > > > as > > > > > > > default > > > > > > > > >>> memory allocator for JDK at our production environment > due > > to > > > > > > > messages > > > > > > > > >> from > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > >>> Moreover, I found several open source projects adopting > > > > jemalloc > > > > > as > > > > > > > > >>> default memory allocator within their images to resolve > > > memory > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > [6]. > > > > > > > > >>> > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > >>> > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > >>> [2] > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > >>> [3] > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > >>> [5] > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> Best > > > > > > > > >>> Yun Tang > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
Hi guys:
refer: https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc code: https://github.com/jemalloc/jemalloc-experiments https://code.woboq.org/userspace/glibc/benchtests/ Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 4:27 PM Yun Tang <[hidden email]> wrote: > Hi > > > Do you see a noticeable performance difference between the two? > @ Stephan Ewen , as we already use jemalloc as default memory allocator in > production, we do not have much experience to compare performace between > glibc and jemalloc. And I did not take a look at the performance difference > when I debug docker OOM. I'll have a try to run benchmark on docker with > different allocators when I have time these days. > > @wang gang, yes, that is what I also observed when I pmap the memory when > using glibc, many memory segments with 64MB size. > > @ Billy Xie, what kind of test case did you use? In my point of view, > compared to who would use more memory in some cases, we should care more > about who would behave under the max limit for most cases. > > Best > Yun Tang > ________________________________ > From: Xie Billy <[hidden email]> > Sent: Friday, October 30, 2020 8:46 > To: [hidden email] <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for debian based Flink docker image > > After long time running test application(>96 hours) , glibc used less > memory than jemalloc , almost half of jemalloc. Can refer this test case. > > Best Regards! > Billy xie(谢志民) > > > On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]> wrote: > > > we also met glic memory leak .it appeared many 64 MB memory segment when > > using pmap command . i am +1 on setting jemalloc as default > > > > 发送自autohome > > ________________________________ > > 发件人: Yu Li <[hidden email]> > > 发送时间: 2020-10-30 00:15:12 > > 收件人: dev <[hidden email]> > > 抄送: [hidden email] <[hidden email]> > > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for > > debian based Flink docker image > > > > Thanks for sharing more thoughts guys. > > > > My experience with malloc libraries are also limited, and as mentioned in > > my previous email [1], my major concern comes from some online > information > > [2] which indicates in some cases jemalloc will consume as much as twice > > the memory than glibc, which is a huge cost from my point of view. > However, > > one may also argue that this post was written in 2015 and things might > have > > been improved a lot ever since. > > > > Nevertheless, as Till and Yun pointed out, I could also see the benefits > of > > setting jemalloc as default, so I don't have strong objections here. But > > this actually is a change of the default memory allocator of our docker > > image (previously glibc is the only choice) which definitely worth a big > > release note. > > > > Best Regards, > > Yu > > > > [1] > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > > [2] https://stackoverflow.com/a/33993215 > > > > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > I think Till has a good point. The default should work well, because > very > > > few users will ever change it. Even on OOMs. > > > So I am slightly in favor of going with jemalloc. > > > > > > @Yun Tang - Do you see a noticeable performance difference between the > > two? > > > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > Hi > > > > > > > > I think Till's view deserves wider discussion. > > > > I want to give my two cents when I debug with Nico on his reported > > > RocksDB > > > > OOM problem. > > > > Jemalloc has the mechanism to profile memory allocation [1] which is > > > > widely used to analysis memory leak. > > > > Once we set jemalloc as default memory allocator, the frequency of > OOM > > > > behavior decreases obviously. > > > > > > > > Considering the OOM killed problem in k8s, change default memory > > > allocator > > > > as jemalloc could be something beneficial. > > > > > > > > [1] > > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > > > Best > > > > Yun Tang > > > > > > > > ________________________________ > > > > From: Till Rohrmann <[hidden email]<mailto: > [hidden email] > > >> > > > > Sent: Thursday, October 29, 2020 18:34 > > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > allocator > > > > for debian based Flink docker image > > > > > > > > Hi Yu, > > > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > > > I honestly don't have a lot of experience with malloc libraries so I > > can > > > > only argue from a general perspective: We know that glibc has some > > > problems > > > > wrt to memory fragmentation which can cause processes to exceed its > > > memory > > > > limit. jemalloc seems to address this problem by avoiding memory > > > > fragmentation. However, it might be the case that jemalloc introduces > > > other > > > > problems. In order to figure this out we would need to collect more > > data > > > > points/experience. So in the end we have to weigh glibc memory > > > > fragmentation against unknown problems in jemalloc. > > > > > > > > Given that choosing the malloc library will be quite an expert > > setting, I > > > > don't expect many Flink users to use it in case they see OOMs. > Hence, I > > > > would be slightly in favour of choosing jemalloc as the default > because > > > it > > > > would allow us to gather feedback from users and solve the existing > > > > problem. In the best case, user's won't run into new problems. If > they > > > do, > > > > then we will have to re-evaluate this decision and might have to > switch > > > the > > > > malloc library back. W/o setting jemalloc as the default, I fear that > > we > > > > will never gather enough feedback to make jemalloc confidently the > > > default. > > > > > > > > What one could argue, though, is that making jemalloc the default so > > > > shortly before the feature freeze limits the timespan we can observe > > how > > > it > > > > behaves before the actual release. Doing such a change at the > beginning > > > of > > > > a release cycle, would allow us to gain more confidence. > > > > > > > > In any way, I believe that we should add a big release note and > > document > > > > what users should do in case they see memory issues (OOM kills, > slower > > > > performance, etc.). > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > > > True, thanks for the reminder Till! > > > > > > > > > > I suggest using glibc malloc as default. On one hand this follows > our > > > old > > > > > behavior (only with glibc malloc support in the image), on the > other > > > > hand I > > > > > believe glibc isn't using jemalloc as its default memory allocator > > for > > > > some > > > > > reason. > > > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email] > > <mailto:[hidden email]>> > > > > wrote: > > > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > > > > > It seems we have reached consensus and there are already enough > > +1s > > > > to > > > > > > move > > > > > > > forward, and suggest @Yun to conclude the discussion directly > if > > > > there > > > > > > are > > > > > > > no objections. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Best Regards, > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > > [hidden email]<mailto:[hidden email]>> > > > > > > wrote: > > > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > > image > > > > then > > > > > > we > > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > > better > > > > > > choice. > > > > > > > > > > We could install jemalloc and only enable it in > LD_PRELOAD > > > when > > > > > > user > > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > > By doing so, we could avoid to create another docker > image > > > tags > > > > > and > > > > > > > > also > > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > Yun Tang > > > > > > > > > > ________________________________ > > > > > > > > > > From: Yang Wang <[hidden email]<mailto: > > [hidden email]>> > > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > > To: dev <[hidden email]<mailto: > [hidden email] > > >> > > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > > > memory > > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > > container > > > > > > > > > environment > > > > > > > > > > in production. I just > > > > > > > > > > have quick question. Could we have both memory > > allocator(e.g. > > > > > > glibc, > > > > > > > > > > jemalloc) in the Flink > > > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> > > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving > > the > > > > > > > > discussion > > > > > > > > > >> Yun! > > > > > > > > > >> > > > > > > > > > >> For the given solutions, I prefer option 1 (supply > another > > > > > > > Dockerfile > > > > > > > > > using > > > > > > > > > >> jemalloc as default memory allocator) because of the > below > > > > > > reasons: > > > > > > > > > >> > > > > > > > > > >> 1. It's hard to say jemalloc is always better than > > ptmalloc > > > > > (glibc > > > > > > > > > malloc), > > > > > > > > > >> or else glibc should have already adopted it as the > > default > > > > > memory > > > > > > > > > >> allocator. And as indicated here [1], in some cases > > jemalloc > > > > > will > > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > > >> > > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > > change > > > > the > > > > > > > > default > > > > > > > > > >> memory allocator to jemalloc and only supply one series > of > > > > > images, > > > > > > > we > > > > > > > > > will > > > > > > > > > >> leave those having better performance with glibc no > other > > > > > choices > > > > > > > but > > > > > > > > > >> staying with old images. In another word, there's a risk > > of > > > > > > > > introducing > > > > > > > > > new > > > > > > > > > >> problems while fixing an existing one if choosing > > option-2. > > > > > > > > > >> > > > > > > > > > >> And there is a third option considering the efforts of > > > > > maintaining > > > > > > > > more > > > > > > > > > >> images if the memory leak issue is not widely observed, > > that > > > > we > > > > > > > could > > > > > > > > > >> document the steps of building Dockerfile with jemalloc > as > > > > > default > > > > > > > > > >> allocator so users could build it when needed, which > > leaves > > > > the > > > > > > > burden > > > > > > > > > to > > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > > >> > > > > > > > > > >> Best Regards, > > > > > > > > > >> Yu > > > > > > > > > >> > > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > > >> > > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang < > [hidden email] > > <mailto:[hidden email]>> > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Hi all > > > > > > > > > >>> > > > > > > > > > >>> Users report they meet serious memory leak when > > submitting > > > > jobs > > > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > > > FLINK-18712[1] > > > > > > > > > ), > > > > > > > > > >>> and I also reproduce this to find this is caused by > > memory > > > > > > > > > fragmentation > > > > > > > > > >> of > > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > > >>> > > > > > > > > > >>> * Quick but not very clean solution to limit the > > > memory > > > > > pool > > > > > > > of > > > > > > > > > >> glibc, > > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > > >>> > > > > > > > > > >>> * More general solution by rebuilding the image to > > > > install > > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > > LD_PRELOAD > > > > > > > > > >>> > > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > > > eventually. > > > > > > > > > Thus, > > > > > > > > > >>> I begin to think whether we should change our > Dockerfile > > to > > > > > adopt > > > > > > > > > >> jemalloc > > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > > >>> > > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > > default > > > > > > > memory > > > > > > > > > >>> allocator, which means Flink needs another two new > image > > > tags > > > > > to > > > > > > > > build > > > > > > > > > >>> docker with jemalloc while default docker still use > > glibc. > > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in > > our > > > > > > existing > > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > > > jemalloc > > > > > > by > > > > > > > > > >> default. > > > > > > > > > >>> I prefer the 2nd option as our company already use > > jemalloc > > > > as > > > > > > > > default > > > > > > > > > >>> memory allocator for JDK at our production environment > > due > > > to > > > > > > > > messages > > > > > > > > > >> from > > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > > >>> Moreover, I found several open source projects adopting > > > > > jemalloc > > > > > > as > > > > > > > > > >>> default memory allocator within their images to resolve > > > > memory > > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > > [6]. > > > > > > > > > >>> > > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > > >>> > > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > > >>> [2] > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > > >>> [3] > > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > > >>> [5] > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> Best > > > > > > > > > >>> Yun Tang > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
Hi @ Billy
I found there existed many benchmark case existed in the two repos, which benchmark case did you run? Best Yun Tang ________________________________ From: Xie Billy <[hidden email]> Sent: Tuesday, November 3, 2020 22:08 To: [hidden email] <[hidden email]> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Hi guys: refer: https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc code: https://github.com/jemalloc/jemalloc-experiments https://code.woboq.org/userspace/glibc/benchtests/ Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 4:27 PM Yun Tang <[hidden email]> wrote: > Hi > > > Do you see a noticeable performance difference between the two? > @ Stephan Ewen , as we already use jemalloc as default memory allocator in > production, we do not have much experience to compare performace between > glibc and jemalloc. And I did not take a look at the performance difference > when I debug docker OOM. I'll have a try to run benchmark on docker with > different allocators when I have time these days. > > @wang gang, yes, that is what I also observed when I pmap the memory when > using glibc, many memory segments with 64MB size. > > @ Billy Xie, what kind of test case did you use? In my point of view, > compared to who would use more memory in some cases, we should care more > about who would behave under the max limit for most cases. > > Best > Yun Tang > ________________________________ > From: Xie Billy <[hidden email]> > Sent: Friday, October 30, 2020 8:46 > To: [hidden email] <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for debian based Flink docker image > > After long time running test application(>96 hours) , glibc used less > memory than jemalloc , almost half of jemalloc. Can refer this test case. > > Best Regards! > Billy xie(谢志民) > > > On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]> wrote: > > > we also met glic memory leak .it appeared many 64 MB memory segment when > > using pmap command . i am +1 on setting jemalloc as default > > > > 发送自autohome > > ________________________________ > > 发件人: Yu Li <[hidden email]> > > 发送时间: 2020-10-30 00:15:12 > > 收件人: dev <[hidden email]> > > 抄送: [hidden email] <[hidden email]> > > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for > > debian based Flink docker image > > > > Thanks for sharing more thoughts guys. > > > > My experience with malloc libraries are also limited, and as mentioned in > > my previous email [1], my major concern comes from some online > information > > [2] which indicates in some cases jemalloc will consume as much as twice > > the memory than glibc, which is a huge cost from my point of view. > However, > > one may also argue that this post was written in 2015 and things might > have > > been improved a lot ever since. > > > > Nevertheless, as Till and Yun pointed out, I could also see the benefits > of > > setting jemalloc as default, so I don't have strong objections here. But > > this actually is a change of the default memory allocator of our docker > > image (previously glibc is the only choice) which definitely worth a big > > release note. > > > > Best Regards, > > Yu > > > > [1] > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > > [2] https://stackoverflow.com/a/33993215 > > > > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > I think Till has a good point. The default should work well, because > very > > > few users will ever change it. Even on OOMs. > > > So I am slightly in favor of going with jemalloc. > > > > > > @Yun Tang - Do you see a noticeable performance difference between the > > two? > > > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > Hi > > > > > > > > I think Till's view deserves wider discussion. > > > > I want to give my two cents when I debug with Nico on his reported > > > RocksDB > > > > OOM problem. > > > > Jemalloc has the mechanism to profile memory allocation [1] which is > > > > widely used to analysis memory leak. > > > > Once we set jemalloc as default memory allocator, the frequency of > OOM > > > > behavior decreases obviously. > > > > > > > > Considering the OOM killed problem in k8s, change default memory > > > allocator > > > > as jemalloc could be something beneficial. > > > > > > > > [1] > > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > > > Best > > > > Yun Tang > > > > > > > > ________________________________ > > > > From: Till Rohrmann <[hidden email]<mailto: > [hidden email] > > >> > > > > Sent: Thursday, October 29, 2020 18:34 > > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > allocator > > > > for debian based Flink docker image > > > > > > > > Hi Yu, > > > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > > > I honestly don't have a lot of experience with malloc libraries so I > > can > > > > only argue from a general perspective: We know that glibc has some > > > problems > > > > wrt to memory fragmentation which can cause processes to exceed its > > > memory > > > > limit. jemalloc seems to address this problem by avoiding memory > > > > fragmentation. However, it might be the case that jemalloc introduces > > > other > > > > problems. In order to figure this out we would need to collect more > > data > > > > points/experience. So in the end we have to weigh glibc memory > > > > fragmentation against unknown problems in jemalloc. > > > > > > > > Given that choosing the malloc library will be quite an expert > > setting, I > > > > don't expect many Flink users to use it in case they see OOMs. > Hence, I > > > > would be slightly in favour of choosing jemalloc as the default > because > > > it > > > > would allow us to gather feedback from users and solve the existing > > > > problem. In the best case, user's won't run into new problems. If > they > > > do, > > > > then we will have to re-evaluate this decision and might have to > switch > > > the > > > > malloc library back. W/o setting jemalloc as the default, I fear that > > we > > > > will never gather enough feedback to make jemalloc confidently the > > > default. > > > > > > > > What one could argue, though, is that making jemalloc the default so > > > > shortly before the feature freeze limits the timespan we can observe > > how > > > it > > > > behaves before the actual release. Doing such a change at the > beginning > > > of > > > > a release cycle, would allow us to gain more confidence. > > > > > > > > In any way, I believe that we should add a big release note and > > document > > > > what users should do in case they see memory issues (OOM kills, > slower > > > > performance, etc.). > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > > > True, thanks for the reminder Till! > > > > > > > > > > I suggest using glibc malloc as default. On one hand this follows > our > > > old > > > > > behavior (only with glibc malloc support in the image), on the > other > > > > hand I > > > > > believe glibc isn't using jemalloc as its default memory allocator > > for > > > > some > > > > > reason. > > > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email] > > <mailto:[hidden email]>> > > > > wrote: > > > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > > > > > It seems we have reached consensus and there are already enough > > +1s > > > > to > > > > > > move > > > > > > > forward, and suggest @Yun to conclude the discussion directly > if > > > > there > > > > > > are > > > > > > > no objections. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Best Regards, > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > > [hidden email]<mailto:[hidden email]>> > > > > > > wrote: > > > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > > image > > > > then > > > > > > we > > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > > better > > > > > > choice. > > > > > > > > > > We could install jemalloc and only enable it in > LD_PRELOAD > > > when > > > > > > user > > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > > By doing so, we could avoid to create another docker > image > > > tags > > > > > and > > > > > > > > also > > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > Yun Tang > > > > > > > > > > ________________________________ > > > > > > > > > > From: Yang Wang <[hidden email]<mailto: > > [hidden email]>> > > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > > To: dev <[hidden email]<mailto: > [hidden email] > > >> > > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > > > memory > > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > > container > > > > > > > > > environment > > > > > > > > > > in production. I just > > > > > > > > > > have quick question. Could we have both memory > > allocator(e.g. > > > > > > glibc, > > > > > > > > > > jemalloc) in the Flink > > > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> > > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving > > the > > > > > > > > discussion > > > > > > > > > >> Yun! > > > > > > > > > >> > > > > > > > > > >> For the given solutions, I prefer option 1 (supply > another > > > > > > > Dockerfile > > > > > > > > > using > > > > > > > > > >> jemalloc as default memory allocator) because of the > below > > > > > > reasons: > > > > > > > > > >> > > > > > > > > > >> 1. It's hard to say jemalloc is always better than > > ptmalloc > > > > > (glibc > > > > > > > > > malloc), > > > > > > > > > >> or else glibc should have already adopted it as the > > default > > > > > memory > > > > > > > > > >> allocator. And as indicated here [1], in some cases > > jemalloc > > > > > will > > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > > >> > > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > > change > > > > the > > > > > > > > default > > > > > > > > > >> memory allocator to jemalloc and only supply one series > of > > > > > images, > > > > > > > we > > > > > > > > > will > > > > > > > > > >> leave those having better performance with glibc no > other > > > > > choices > > > > > > > but > > > > > > > > > >> staying with old images. In another word, there's a risk > > of > > > > > > > > introducing > > > > > > > > > new > > > > > > > > > >> problems while fixing an existing one if choosing > > option-2. > > > > > > > > > >> > > > > > > > > > >> And there is a third option considering the efforts of > > > > > maintaining > > > > > > > > more > > > > > > > > > >> images if the memory leak issue is not widely observed, > > that > > > > we > > > > > > > could > > > > > > > > > >> document the steps of building Dockerfile with jemalloc > as > > > > > default > > > > > > > > > >> allocator so users could build it when needed, which > > leaves > > > > the > > > > > > > burden > > > > > > > > > to > > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > > >> > > > > > > > > > >> Best Regards, > > > > > > > > > >> Yu > > > > > > > > > >> > > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > > >> > > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang < > [hidden email] > > <mailto:[hidden email]>> > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Hi all > > > > > > > > > >>> > > > > > > > > > >>> Users report they meet serious memory leak when > > submitting > > > > jobs > > > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > > > FLINK-18712[1] > > > > > > > > > ), > > > > > > > > > >>> and I also reproduce this to find this is caused by > > memory > > > > > > > > > fragmentation > > > > > > > > > >> of > > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > > >>> > > > > > > > > > >>> * Quick but not very clean solution to limit the > > > memory > > > > > pool > > > > > > > of > > > > > > > > > >> glibc, > > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > > >>> > > > > > > > > > >>> * More general solution by rebuilding the image to > > > > install > > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > > LD_PRELOAD > > > > > > > > > >>> > > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > > > eventually. > > > > > > > > > Thus, > > > > > > > > > >>> I begin to think whether we should change our > Dockerfile > > to > > > > > adopt > > > > > > > > > >> jemalloc > > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > > >>> > > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > > default > > > > > > > memory > > > > > > > > > >>> allocator, which means Flink needs another two new > image > > > tags > > > > > to > > > > > > > > build > > > > > > > > > >>> docker with jemalloc while default docker still use > > glibc. > > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in > > our > > > > > > existing > > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > > > jemalloc > > > > > > by > > > > > > > > > >> default. > > > > > > > > > >>> I prefer the 2nd option as our company already use > > jemalloc > > > > as > > > > > > > > default > > > > > > > > > >>> memory allocator for JDK at our production environment > > due > > > to > > > > > > > > messages > > > > > > > > > >> from > > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > > >>> Moreover, I found several open source projects adopting > > > > > jemalloc > > > > > > as > > > > > > > > > >>> default memory allocator within their images to resolve > > > > memory > > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > > [6]. > > > > > > > > > >>> > > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > > >>> > > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > > >>> [2] > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > > >>> [3] > > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > > >>> [5] > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> Best > > > > > > > > > >>> Yun Tang > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
|
|
For your information,
I executed the flink-benchmarks [1] twice within docker container based on flink-1.11.1 scala_2.11-java8-debian image [2], from the results of state benchmarks, I cannot see obvious performance changes whether to use jemalloc as memory allocator: RocksDB keyed state backend ops/ms glibc jemalloc comparison ListStateBenchmark.listAdd 537.613 549.926 2.29% ListStateBenchmark.listAddAll 301.764 295.51 -2.07% ListStateBenchmark.listAppend 521.32 522.614 0.25% ListStateBenchmark.listGet 139.19 141.321 1.53% ListStateBenchmark.listGetAndIterate 139.685 141.871 1.56% ListStateBenchmark.listUpdate 534.785 559.509 4.62% MapStateBenchmark.mapAdd 469.748 475.241 1.17% MapStateBenchmark.mapContains 51.481 52.188 1.37% MapStateBenchmark.mapEntries 352.439 357.951 1.56% MapStateBenchmark.mapGet 51.903 52.065 0.31% MapStateBenchmark.mapIsEmpty 47.38 48.16 1.65% MapStateBenchmark.mapIterator 351.41 357.412 1.71% MapStateBenchmark.mapKeys 361.339 359.773 -0.43% MapStateBenchmark.mapPutAll 117.067 111.842 -4.46% MapStateBenchmark.mapRemove 497.361 499.771 0.48% MapStateBenchmark.mapUpdate 464.865 463.501 -0.29% MapStateBenchmark.mapValues 350.942 358.64 2.19% ValueStateBenchmark.valueAdd 475.55 462.627 -2.72% ValueStateBenchmark.valueGet 713.389 729.126 2.21% ValueStateBenchmark.valueUpdate 476.373 482.183 1.22% Heap keyed state backend ops/ms glibc jemalloc comparison ListStateBenchmark.listAdd 6711 6614.719 -1.43% ListStateBenchmark.listAddAll 700.429 713.487 1.86% ListStateBenchmark.listAppend 2841.068 2848.416 0.26% ListStateBenchmark.listGet 2863.704 2835.862 -0.97% ListStateBenchmark.listGetAndIterate 2790.001 2787.145 -0.10% ListStateBenchmark.listUpdate 2802.287 2802.626 0.01% MapStateBenchmark.mapAdd 1939.755 1950.759 0.57% MapStateBenchmark.mapContains 1914.49 1943.529 1.52% MapStateBenchmark.mapEntries 11836.215 11836.673 0.00% MapStateBenchmark.mapGet 1753.817 1756.643 0.16% MapStateBenchmark.mapIsEmpty 2980.299 2960.752 -0.66% MapStateBenchmark.mapIterator 11151.177 11123.037 -0.25% MapStateBenchmark.mapKeys 12956.381 12778.626 -1.37% MapStateBenchmark.mapPutAll 1253.269 1247.15 -0.49% MapStateBenchmark.mapRemove 2594.152 2575.233 -0.73% MapStateBenchmark.mapUpdate 1865.745 1880.573 0.79% MapStateBenchmark.mapValues 12546.359 12473.223 -0.58% ValueStateBenchmark.valueAdd 3947.228 3932.003 -0.39% ValueStateBenchmark.valueGet 3887.003 3863.13 -0.61% ValueStateBenchmark.valueUpdate 3978.979 3973.183 -0.15% [1] https://github.com/apache/flink-benchmarks [2] https://github.com/apache/flink-docker/tree/master/1.11/scala_2.11-java8-debian Best Yun Tang ________________________________ From: Yun Tang <[hidden email]> Sent: Wednesday, November 4, 2020 15:41 To: [hidden email] <[hidden email]> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Hi @ Billy I found there existed many benchmark case existed in the two repos, which benchmark case did you run? Best Yun Tang ________________________________ From: Xie Billy <[hidden email]> Sent: Tuesday, November 3, 2020 22:08 To: [hidden email] <[hidden email]> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image Hi guys: refer: https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc code: https://github.com/jemalloc/jemalloc-experiments https://code.woboq.org/userspace/glibc/benchtests/ Best Regards! Billy xie(谢志民) On Fri, Oct 30, 2020 at 4:27 PM Yun Tang <[hidden email]> wrote: > Hi > > > Do you see a noticeable performance difference between the two? > @ Stephan Ewen , as we already use jemalloc as default memory allocator in > production, we do not have much experience to compare performace between > glibc and jemalloc. And I did not take a look at the performance difference > when I debug docker OOM. I'll have a try to run benchmark on docker with > different allocators when I have time these days. > > @wang gang, yes, that is what I also observed when I pmap the memory when > using glibc, many memory segments with 64MB size. > > @ Billy Xie, what kind of test case did you use? In my point of view, > compared to who would use more memory in some cases, we should care more > about who would behave under the max limit for most cases. > > Best > Yun Tang > ________________________________ > From: Xie Billy <[hidden email]> > Sent: Friday, October 30, 2020 8:46 > To: [hidden email] <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for debian based Flink docker image > > After long time running test application(>96 hours) , glibc used less > memory than jemalloc , almost half of jemalloc. Can refer this test case. > > Best Regards! > Billy xie(谢志民) > > > On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]> wrote: > > > we also met glic memory leak .it appeared many 64 MB memory segment when > > using pmap command . i am +1 on setting jemalloc as default > > > > 发送自autohome > > ________________________________ > > 发件人: Yu Li <[hidden email]> > > 发送时间: 2020-10-30 00:15:12 > > 收件人: dev <[hidden email]> > > 抄送: [hidden email] <[hidden email]> > > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for > > debian based Flink docker image > > > > Thanks for sharing more thoughts guys. > > > > My experience with malloc libraries are also limited, and as mentioned in > > my previous email [1], my major concern comes from some online > information > > [2] which indicates in some cases jemalloc will consume as much as twice > > the memory than glibc, which is a huge cost from my point of view. > However, > > one may also argue that this post was written in 2015 and things might > have > > been improved a lot ever since. > > > > Nevertheless, as Till and Yun pointed out, I could also see the benefits > of > > setting jemalloc as default, so I don't have strong objections here. But > > this actually is a change of the default memory allocator of our docker > > image (previously glibc is the only choice) which definitely worth a big > > release note. > > > > Best Regards, > > Yu > > > > [1] > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > > [2] https://stackoverflow.com/a/33993215 > > > > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > I think Till has a good point. The default should work well, because > very > > > few users will ever change it. Even on OOMs. > > > So I am slightly in favor of going with jemalloc. > > > > > > @Yun Tang - Do you see a noticeable performance difference between the > > two? > > > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > Hi > > > > > > > > I think Till's view deserves wider discussion. > > > > I want to give my two cents when I debug with Nico on his reported > > > RocksDB > > > > OOM problem. > > > > Jemalloc has the mechanism to profile memory allocation [1] which is > > > > widely used to analysis memory leak. > > > > Once we set jemalloc as default memory allocator, the frequency of > OOM > > > > behavior decreases obviously. > > > > > > > > Considering the OOM killed problem in k8s, change default memory > > > allocator > > > > as jemalloc could be something beneficial. > > > > > > > > [1] > > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > > > Best > > > > Yun Tang > > > > > > > > ________________________________ > > > > From: Till Rohrmann <[hidden email]<mailto: > [hidden email] > > >> > > > > Sent: Thursday, October 29, 2020 18:34 > > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > allocator > > > > for debian based Flink docker image > > > > > > > > Hi Yu, > > > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > > > I honestly don't have a lot of experience with malloc libraries so I > > can > > > > only argue from a general perspective: We know that glibc has some > > > problems > > > > wrt to memory fragmentation which can cause processes to exceed its > > > memory > > > > limit. jemalloc seems to address this problem by avoiding memory > > > > fragmentation. However, it might be the case that jemalloc introduces > > > other > > > > problems. In order to figure this out we would need to collect more > > data > > > > points/experience. So in the end we have to weigh glibc memory > > > > fragmentation against unknown problems in jemalloc. > > > > > > > > Given that choosing the malloc library will be quite an expert > > setting, I > > > > don't expect many Flink users to use it in case they see OOMs. > Hence, I > > > > would be slightly in favour of choosing jemalloc as the default > because > > > it > > > > would allow us to gather feedback from users and solve the existing > > > > problem. In the best case, user's won't run into new problems. If > they > > > do, > > > > then we will have to re-evaluate this decision and might have to > switch > > > the > > > > malloc library back. W/o setting jemalloc as the default, I fear that > > we > > > > will never gather enough feedback to make jemalloc confidently the > > > default. > > > > > > > > What one could argue, though, is that making jemalloc the default so > > > > shortly before the feature freeze limits the timespan we can observe > > how > > > it > > > > behaves before the actual release. Doing such a change at the > beginning > > > of > > > > a release cycle, would allow us to gain more confidence. > > > > > > > > In any way, I believe that we should add a big release note and > > document > > > > what users should do in case they see memory issues (OOM kills, > slower > > > > performance, etc.). > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > > > True, thanks for the reminder Till! > > > > > > > > > > I suggest using glibc malloc as default. On one hand this follows > our > > > old > > > > > behavior (only with glibc malloc support in the image), on the > other > > > > hand I > > > > > believe glibc isn't using jemalloc as its default memory allocator > > for > > > > some > > > > > reason. > > > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <[hidden email] > > <mailto:[hidden email]>> > > > > wrote: > > > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]<mailto: > > [hidden email]>> wrote: > > > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker image. > > > > > > > > > > > > > > It seems we have reached consensus and there are already enough > > +1s > > > > to > > > > > > move > > > > > > > forward, and suggest @Yun to conclude the discussion directly > if > > > > there > > > > > > are > > > > > > > no objections. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Best Regards, > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > > [hidden email]<mailto:[hidden email]>> > > > > > > wrote: > > > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > > image > > > > then > > > > > > we > > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > > better > > > > > > choice. > > > > > > > > > > We could install jemalloc and only enable it in > LD_PRELOAD > > > when > > > > > > user > > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > > By doing so, we could avoid to create another docker > image > > > tags > > > > > and > > > > > > > > also > > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > Yun Tang > > > > > > > > > > ________________________________ > > > > > > > > > > From: Yang Wang <[hidden email]<mailto: > > [hidden email]>> > > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > > To: dev <[hidden email]<mailto: > [hidden email] > > >> > > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default > > > memory > > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > > container > > > > > > > > > environment > > > > > > > > > > in production. I just > > > > > > > > > > have quick question. Could we have both memory > > allocator(e.g. > > > > > > glibc, > > > > > > > > > > jemalloc) in the Flink > > > > > > > > > > official image and enable a specific one by setting ENV? > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> > > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and driving > > the > > > > > > > > discussion > > > > > > > > > >> Yun! > > > > > > > > > >> > > > > > > > > > >> For the given solutions, I prefer option 1 (supply > another > > > > > > > Dockerfile > > > > > > > > > using > > > > > > > > > >> jemalloc as default memory allocator) because of the > below > > > > > > reasons: > > > > > > > > > >> > > > > > > > > > >> 1. It's hard to say jemalloc is always better than > > ptmalloc > > > > > (glibc > > > > > > > > > malloc), > > > > > > > > > >> or else glibc should have already adopted it as the > > default > > > > > memory > > > > > > > > > >> allocator. And as indicated here [1], in some cases > > jemalloc > > > > > will > > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > > >> > > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > > change > > > > the > > > > > > > > default > > > > > > > > > >> memory allocator to jemalloc and only supply one series > of > > > > > images, > > > > > > > we > > > > > > > > > will > > > > > > > > > >> leave those having better performance with glibc no > other > > > > > choices > > > > > > > but > > > > > > > > > >> staying with old images. In another word, there's a risk > > of > > > > > > > > introducing > > > > > > > > > new > > > > > > > > > >> problems while fixing an existing one if choosing > > option-2. > > > > > > > > > >> > > > > > > > > > >> And there is a third option considering the efforts of > > > > > maintaining > > > > > > > > more > > > > > > > > > >> images if the memory leak issue is not widely observed, > > that > > > > we > > > > > > > could > > > > > > > > > >> document the steps of building Dockerfile with jemalloc > as > > > > > default > > > > > > > > > >> allocator so users could build it when needed, which > > leaves > > > > the > > > > > > > burden > > > > > > > > > to > > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > > >> > > > > > > > > > >> Best Regards, > > > > > > > > > >> Yu > > > > > > > > > >> > > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > > >> > > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang < > [hidden email] > > <mailto:[hidden email]>> > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Hi all > > > > > > > > > >>> > > > > > > > > > >>> Users report they meet serious memory leak when > > submitting > > > > jobs > > > > > > > > > >>> continously in session mode within k8s (please refer to > > > > > > > > FLINK-18712[1] > > > > > > > > > ), > > > > > > > > > >>> and I also reproduce this to find this is caused by > > memory > > > > > > > > > fragmentation > > > > > > > > > >> of > > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > > >>> > > > > > > > > > >>> * Quick but not very clean solution to limit the > > > memory > > > > > pool > > > > > > > of > > > > > > > > > >> glibc, > > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > > >>> > > > > > > > > > >>> * More general solution by rebuilding the image to > > > > install > > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > > LD_PRELOAD > > > > > > > > > >>> > > > > > > > > > >>> The reporter adopted the 2nd solution to fix this issue > > > > > > eventually. > > > > > > > > > Thus, > > > > > > > > > >>> I begin to think whether we should change our > Dockerfile > > to > > > > > adopt > > > > > > > > > >> jemalloc > > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > > >>> > > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > > default > > > > > > > memory > > > > > > > > > >>> allocator, which means Flink needs another two new > image > > > tags > > > > > to > > > > > > > > build > > > > > > > > > >>> docker with jemalloc while default docker still use > > glibc. > > > > > > > > > >>> 2. Set the default memory allocator as jemalloc in > > our > > > > > > existing > > > > > > > > > >>> Dockerfiles, which means Flink offer docker image with > > > > jemalloc > > > > > > by > > > > > > > > > >> default. > > > > > > > > > >>> I prefer the 2nd option as our company already use > > jemalloc > > > > as > > > > > > > > default > > > > > > > > > >>> memory allocator for JDK at our production environment > > due > > > to > > > > > > > > messages > > > > > > > > > >> from > > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > > >>> Moreover, I found several open source projects adopting > > > > > jemalloc > > > > > > as > > > > > > > > > >>> default memory allocator within their images to resolve > > > > memory > > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > > [6]. > > > > > > > > > >>> > > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > > >>> > > > > > > > > > >>> [1] https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > > >>> [2] > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > > >>> [3] > > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > > >>> [4] https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > > >>> [5] > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > > >>> [6] https://github.com/home-assistant/core/pull/33237 > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> Best > > > > > > > > > >>> Yun Tang > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image
|
|
Thanks for running the benchmarks Yun Tang. From my perspective I think it
would be fine to switch to jemalloc and offer an option to switch it back if needed. Cheers, Till On Thu, Nov 5, 2020 at 1:32 PM Yun Tang <[hidden email]> wrote: > For your information, > > I executed the flink-benchmarks [1] twice within docker container based on > flink-1.11.1 scala_2.11-java8-debian image [2], from the results of state > benchmarks, I cannot see obvious performance changes whether to use > jemalloc as memory allocator: > > RocksDB keyed state backend ops/ms > glibc jemalloc comparison > ListStateBenchmark.listAdd 537.613 549.926 2.29% > ListStateBenchmark.listAddAll 301.764 295.51 -2.07% > ListStateBenchmark.listAppend 521.32 522.614 0.25% > ListStateBenchmark.listGet 139.19 141.321 1.53% > ListStateBenchmark.listGetAndIterate 139.685 141.871 1.56% > ListStateBenchmark.listUpdate 534.785 559.509 4.62% > MapStateBenchmark.mapAdd 469.748 475.241 1.17% > MapStateBenchmark.mapContains 51.481 52.188 1.37% > MapStateBenchmark.mapEntries 352.439 357.951 1.56% > MapStateBenchmark.mapGet 51.903 52.065 0.31% > MapStateBenchmark.mapIsEmpty 47.38 48.16 1.65% > MapStateBenchmark.mapIterator 351.41 357.412 1.71% > MapStateBenchmark.mapKeys 361.339 359.773 -0.43% > MapStateBenchmark.mapPutAll 117.067 111.842 -4.46% > MapStateBenchmark.mapRemove 497.361 499.771 0.48% > MapStateBenchmark.mapUpdate 464.865 463.501 -0.29% > MapStateBenchmark.mapValues 350.942 358.64 2.19% > ValueStateBenchmark.valueAdd 475.55 462.627 -2.72% > ValueStateBenchmark.valueGet 713.389 729.126 2.21% > ValueStateBenchmark.valueUpdate 476.373 482.183 1.22% > > Heap keyed state backend ops/ms > glibc jemalloc comparison > ListStateBenchmark.listAdd 6711 6614.719 -1.43% > ListStateBenchmark.listAddAll 700.429 713.487 1.86% > ListStateBenchmark.listAppend 2841.068 2848.416 > 0.26% > ListStateBenchmark.listGet 2863.704 2835.862 > -0.97% > ListStateBenchmark.listGetAndIterate 2790.001 2787.145 > -0.10% > ListStateBenchmark.listUpdate 2802.287 2802.626 > 0.01% > MapStateBenchmark.mapAdd 1939.755 1950.759 > 0.57% > MapStateBenchmark.mapContains 1914.49 1943.529 1.52% > MapStateBenchmark.mapEntries 11836.215 11836.673 > 0.00% > MapStateBenchmark.mapGet 1753.817 1756.643 > 0.16% > MapStateBenchmark.mapIsEmpty 2980.299 2960.752 > -0.66% > MapStateBenchmark.mapIterator 11151.177 11123.037 > -0.25% > MapStateBenchmark.mapKeys 12956.381 12778.626 > -1.37% > MapStateBenchmark.mapPutAll 1253.269 1247.15 -0.49% > MapStateBenchmark.mapRemove 2594.152 2575.233 > -0.73% > MapStateBenchmark.mapUpdate 1865.745 1880.573 > 0.79% > MapStateBenchmark.mapValues 12546.359 12473.223 > -0.58% > ValueStateBenchmark.valueAdd 3947.228 3932.003 > -0.39% > ValueStateBenchmark.valueGet 3887.003 3863.13 -0.61% > ValueStateBenchmark.valueUpdate 3978.979 3973.183 > -0.15% > > > [1] https://github.com/apache/flink-benchmarks > [2] > https://github.com/apache/flink-docker/tree/master/1.11/scala_2.11-java8-debian > > > Best > Yun Tang > > ________________________________ > From: Yun Tang <[hidden email]> > Sent: Wednesday, November 4, 2020 15:41 > To: [hidden email] <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for debian based Flink docker image > > Hi @ Billy > > I found there existed many benchmark case existed in the two repos, which > benchmark case did you run? > > > Best > Yun Tang > ________________________________ > From: Xie Billy <[hidden email]> > Sent: Tuesday, November 3, 2020 22:08 > To: [hidden email] <[hidden email]> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for debian based Flink docker image > > Hi guys: > refer: > > > https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc > code: https://github.com/jemalloc/jemalloc-experiments > https://code.woboq.org/userspace/glibc/benchtests/ > > > > > Best Regards! > Billy xie(谢志民) > > > On Fri, Oct 30, 2020 at 4:27 PM Yun Tang <[hidden email]> wrote: > > > Hi > > > > > Do you see a noticeable performance difference between the two? > > @ Stephan Ewen , as we already use jemalloc as default memory allocator > in > > production, we do not have much experience to compare performace between > > glibc and jemalloc. And I did not take a look at the performance > difference > > when I debug docker OOM. I'll have a try to run benchmark on docker with > > different allocators when I have time these days. > > > > @wang gang, yes, that is what I also observed when I pmap the memory when > > using glibc, many memory segments with 64MB size. > > > > @ Billy Xie, what kind of test case did you use? In my point of view, > > compared to who would use more memory in some cases, we should care more > > about who would behave under the max limit for most cases. > > > > Best > > Yun Tang > > ________________________________ > > From: Xie Billy <[hidden email]> > > Sent: Friday, October 30, 2020 8:46 > > To: [hidden email] <[hidden email]> > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > > for debian based Flink docker image > > > > After long time running test application(>96 hours) , glibc used less > > memory than jemalloc , almost half of jemalloc. Can refer this test case. > > > > Best Regards! > > Billy xie(谢志民) > > > > > > On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]> > wrote: > > > > > we also met glic memory leak .it appeared many 64 MB memory segment > when > > > using pmap command . i am +1 on setting jemalloc as default > > > > > > 发送自autohome > > > ________________________________ > > > 发件人: Yu Li <[hidden email]> > > > 发送时间: 2020-10-30 00:15:12 > > > 收件人: dev <[hidden email]> > > > 抄送: [hidden email] <[hidden email]> > > > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator > for > > > debian based Flink docker image > > > > > > Thanks for sharing more thoughts guys. > > > > > > My experience with malloc libraries are also limited, and as mentioned > in > > > my previous email [1], my major concern comes from some online > > information > > > [2] which indicates in some cases jemalloc will consume as much as > twice > > > the memory than glibc, which is a huge cost from my point of view. > > However, > > > one may also argue that this post was written in 2015 and things might > > have > > > been improved a lot ever since. > > > > > > Nevertheless, as Till and Yun pointed out, I could also see the > benefits > > of > > > setting jemalloc as default, so I don't have strong objections here. > But > > > this actually is a change of the default memory allocator of our docker > > > image (previously glibc is the only choice) which definitely worth a > big > > > release note. > > > > > > Best Regards, > > > Yu > > > > > > [1] > > > > > > > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html > > > [2] https://stackoverflow.com/a/33993215 > > > > > > > > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto: > > > [hidden email]>> wrote: > > > > > > > I think Till has a good point. The default should work well, because > > very > > > > few users will ever change it. Even on OOMs. > > > > So I am slightly in favor of going with jemalloc. > > > > > > > > @Yun Tang - Do you see a noticeable performance difference between > the > > > two? > > > > > > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto: > > > [hidden email]>> wrote: > > > > > > > > > Hi > > > > > > > > > > I think Till's view deserves wider discussion. > > > > > I want to give my two cents when I debug with Nico on his reported > > > > RocksDB > > > > > OOM problem. > > > > > Jemalloc has the mechanism to profile memory allocation [1] which > is > > > > > widely used to analysis memory leak. > > > > > Once we set jemalloc as default memory allocator, the frequency of > > OOM > > > > > behavior decreases obviously. > > > > > > > > > > Considering the OOM killed problem in k8s, change default memory > > > > allocator > > > > > as jemalloc could be something beneficial. > > > > > > > > > > [1] > > > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling > > > > > > > > > > Best > > > > > Yun Tang > > > > > > > > > > ________________________________ > > > > > From: Till Rohrmann <[hidden email]<mailto: > > [hidden email] > > > >> > > > > > Sent: Thursday, October 29, 2020 18:34 > > > > > To: dev <[hidden email]<mailto:[hidden email]>> > > > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>> > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory > > > allocator > > > > > for debian based Flink docker image > > > > > > > > > > Hi Yu, > > > > > > > > > > I see your point why you are in favour of continuing using glibc. > > > > > > > > > > I honestly don't have a lot of experience with malloc libraries so > I > > > can > > > > > only argue from a general perspective: We know that glibc has some > > > > problems > > > > > wrt to memory fragmentation which can cause processes to exceed its > > > > memory > > > > > limit. jemalloc seems to address this problem by avoiding memory > > > > > fragmentation. However, it might be the case that jemalloc > introduces > > > > other > > > > > problems. In order to figure this out we would need to collect more > > > data > > > > > points/experience. So in the end we have to weigh glibc memory > > > > > fragmentation against unknown problems in jemalloc. > > > > > > > > > > Given that choosing the malloc library will be quite an expert > > > setting, I > > > > > don't expect many Flink users to use it in case they see OOMs. > > Hence, I > > > > > would be slightly in favour of choosing jemalloc as the default > > because > > > > it > > > > > would allow us to gather feedback from users and solve the existing > > > > > problem. In the best case, user's won't run into new problems. If > > they > > > > do, > > > > > then we will have to re-evaluate this decision and might have to > > switch > > > > the > > > > > malloc library back. W/o setting jemalloc as the default, I fear > that > > > we > > > > > will never gather enough feedback to make jemalloc confidently the > > > > default. > > > > > > > > > > What one could argue, though, is that making jemalloc the default > so > > > > > shortly before the feature freeze limits the timespan we can > observe > > > how > > > > it > > > > > behaves before the actual release. Doing such a change at the > > beginning > > > > of > > > > > a release cycle, would allow us to gain more confidence. > > > > > > > > > > In any way, I believe that we should add a big release note and > > > document > > > > > what users should do in case they see memory issues (OOM kills, > > slower > > > > > performance, etc.). > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto: > > > [hidden email]>> wrote: > > > > > > > > > > > True, thanks for the reminder Till! > > > > > > > > > > > > I suggest using glibc malloc as default. On one hand this follows > > our > > > > old > > > > > > behavior (only with glibc malloc support in the image), on the > > other > > > > > hand I > > > > > > believe glibc isn't using jemalloc as its default memory > allocator > > > for > > > > > some > > > > > > reason. > > > > > > > > > > > > Please let me know your thoughts. Thanks. > > > > > > > > > > > > Best Regards, > > > > > > Yu > > > > > > > > > > > > > > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann < > [hidden email] > > > <mailto:[hidden email]>> > > > > > wrote: > > > > > > > > > > > > > The only question left would be what will be the default value? > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email] > <mailto: > > > [hidden email]>> wrote: > > > > > > > > > > > > > > > I'm also +1 on making it configurable in the same Docker > image. > > > > > > > > > > > > > > > > It seems we have reached consensus and there are already > enough > > > +1s > > > > > to > > > > > > > move > > > > > > > > forward, and suggest @Yun to conclude the discussion directly > > if > > > > > there > > > > > > > are > > > > > > > > no objections. > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann < > > > [hidden email]<mailto:[hidden email]>> > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 for making it configurable in the same Docker image. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler < > > > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > If it is possible to support both allocators in a single > > > image > > > > > then > > > > > > > we > > > > > > > > > > should definitely go with that option. > > > > > > > > > > > > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote: > > > > > > > > > > > Thanks for Yang's suggestion. I think this could be a > > > better > > > > > > > choice. > > > > > > > > > > > We could install jemalloc and only enable it in > > LD_PRELOAD > > > > when > > > > > > > user > > > > > > > > > > pass specific configuration for docker-entrypoint.sh. > > > > > > > > > > > By doing so, we could avoid to create another docker > > image > > > > tags > > > > > > and > > > > > > > > > also > > > > > > > > > > offer ability to reduce memory fragmentation problem. > > > > > > > > > > > > > > > > > > > > > > Does anyone else have other ideas? > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > > Yun Tang > > > > > > > > > > > ________________________________ > > > > > > > > > > > From: Yang Wang <[hidden email]<mailto: > > > [hidden email]>> > > > > > > > > > > > Sent: Thursday, October 15, 2020 14:59 > > > > > > > > > > > To: dev <[hidden email]<mailto: > > [hidden email] > > > >> > > > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as > default > > > > memory > > > > > > > > > > allocator for debian based Flink docker image > > > > > > > > > > > > > > > > > > > > > > Thanks Yun Tang for starting this discussion. > > > > > > > > > > > > > > > > > > > > > > I think this is very important when deploy Flink with > > > > container > > > > > > > > > > environment > > > > > > > > > > > in production. I just > > > > > > > > > > > have quick question. Could we have both memory > > > allocator(e.g. > > > > > > > glibc, > > > > > > > > > > > jemalloc) in the Flink > > > > > > > > > > > official image and enable a specific one by setting > ENV? > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>> > > > 于2020年10月14日周三 下午12:23写道： > > > > > > > > > > > > > > > > > > > > > >> Thanks for debugging and resolving the issue and > driving > > > the > > > > > > > > > discussion > > > > > > > > > > >> Yun! > > > > > > > > > > >> > > > > > > > > > > >> For the given solutions, I prefer option 1 (supply > > another > > > > > > > > Dockerfile > > > > > > > > > > using > > > > > > > > > > >> jemalloc as default memory allocator) because of the > > below > > > > > > > reasons: > > > > > > > > > > >> > > > > > > > > > > >> 1. It's hard to say jemalloc is always better than > > > ptmalloc > > > > > > (glibc > > > > > > > > > > malloc), > > > > > > > > > > >> or else glibc should have already adopted it as the > > > default > > > > > > memory > > > > > > > > > > >> allocator. And as indicated here [1], in some cases > > > jemalloc > > > > > > will > > > > > > > > > > >> consume as much as twice the memory than glibc > > > > > > > > > > >> > > > > > > > > > > >> 2. All existing Flink docker images use glibc, if we > > > change > > > > > the > > > > > > > > > default > > > > > > > > > > >> memory allocator to jemalloc and only supply one > series > > of > > > > > > images, > > > > > > > > we > > > > > > > > > > will > > > > > > > > > > >> leave those having better performance with glibc no > > other > > > > > > choices > > > > > > > > but > > > > > > > > > > >> staying with old images. In another word, there's a > risk > > > of > > > > > > > > > introducing > > > > > > > > > > new > > > > > > > > > > >> problems while fixing an existing one if choosing > > > option-2. > > > > > > > > > > >> > > > > > > > > > > >> And there is a third option considering the efforts of > > > > > > maintaining > > > > > > > > > more > > > > > > > > > > >> images if the memory leak issue is not widely > observed, > > > that > > > > > we > > > > > > > > could > > > > > > > > > > >> document the steps of building Dockerfile with > jemalloc > > as > > > > > > default > > > > > > > > > > >> allocator so users could build it when needed, which > > > leaves > > > > > the > > > > > > > > burden > > > > > > > > > > to > > > > > > > > > > >> our users so for me it's not the best option. > > > > > > > > > > >> > > > > > > > > > > >> Best Regards, > > > > > > > > > > >> Yu > > > > > > > > > > >> > > > > > > > > > > >> [1] https://stackoverflow.com/a/33993215 > > > > > > > > > > >> > > > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang < > > [hidden email] > > > <mailto:[hidden email]>> > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > >>> Hi all > > > > > > > > > > >>> > > > > > > > > > > >>> Users report they meet serious memory leak when > > > submitting > > > > > jobs > > > > > > > > > > >>> continously in session mode within k8s (please refer > to > > > > > > > > > FLINK-18712[1] > > > > > > > > > > ), > > > > > > > > > > >>> and I also reproduce this to find this is caused by > > > memory > > > > > > > > > > fragmentation > > > > > > > > > > >> of > > > > > > > > > > >>> glibc [2][3] and provide solutions to fix this: > > > > > > > > > > >>> > > > > > > > > > > >>> * Quick but not very clean solution to limit the > > > > memory > > > > > > pool > > > > > > > > of > > > > > > > > > > >> glibc, > > > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2 > > > > > > > > > > >>> > > > > > > > > > > >>> * More general solution by rebuilding the image > to > > > > > install > > > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to > > > LD_PRELOAD > > > > > > > > > > >>> > > > > > > > > > > >>> The reporter adopted the 2nd solution to fix this > issue > > > > > > > eventually. > > > > > > > > > > Thus, > > > > > > > > > > >>> I begin to think whether we should change our > > Dockerfile > > > to > > > > > > adopt > > > > > > > > > > >> jemalloc > > > > > > > > > > >>> as default memory allocator [4]. > > > > > > > > > > >>> From my point of view, we have two choices: > > > > > > > > > > >>> > > > > > > > > > > >>> 1. Introduce another Dockerfile using jemalloc as > > > > default > > > > > > > > memory > > > > > > > > > > >>> allocator, which means Flink needs another two new > > image > > > > tags > > > > > > to > > > > > > > > > build > > > > > > > > > > >>> docker with jemalloc while default docker still use > > > glibc. > > > > > > > > > > >>> 2. Set the default memory allocator as jemalloc > in > > > our > > > > > > > existing > > > > > > > > > > >>> Dockerfiles, which means Flink offer docker image > with > > > > > jemalloc > > > > > > > by > > > > > > > > > > >> default. > > > > > > > > > > >>> I prefer the 2nd option as our company already use > > > jemalloc > > > > > as > > > > > > > > > default > > > > > > > > > > >>> memory allocator for JDK at our production > environment > > > due > > > > to > > > > > > > > > messages > > > > > > > > > > >> from > > > > > > > > > > >>> os team warning of glibc's memory fragmentation. > > > > > > > > > > >>> Moreover, I found several open source projects > adopting > > > > > > jemalloc > > > > > > > as > > > > > > > > > > >>> default memory allocator within their images to > resolve > > > > > memory > > > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant > > > [6]. > > > > > > > > > > >>> > > > > > > > > > > >>> What do you guys think of this issue? > > > > > > > > > > >>> > > > > > > > > > > >>> [1] > https://issues.apache.org/jira/browse/FLINK-18712 > > > > > > > > > > >>> [2] > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc > > > > > > > > > > >>> [3] > > > https://sourceware.org/bugzilla/show_bug.cgi?id=15321 > > > > > > > > > > >>> [4] > https://issues.apache.org/jira/browse/FLINK-19125 > > > > > > > > > > >>> [5] > > > > > > > > > > >>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux > > > > > > > > > > >>> [6] > https://github.com/home-assistant/core/pull/33237 > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> Best > > > > > > > > > > >>> Yun Tang > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |