[DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

[Yun Tang]

I think we have given enough thoughts but not clear selections currently. I'll create a VOTE thread to choose which memory allocator as default. Hope you could jump in to provide your choices.

Best
Yun Tang
________________________________
From: Till Rohrmann <[hidden email]>
Sent: Friday, November 6, 2020 16:48
To: dev <[hidden email]>
Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator for debian based Flink docker image

Thanks for running the benchmarks Yun Tang. From my perspective I think it
would be fine to switch to jemalloc and offer an option to switch it back
if needed.

Cheers,
Till

On Thu, Nov 5, 2020 at 1:32 PM Yun Tang <[hidden email]> wrote:

> For your information,
>
> I executed the flink-benchmarks [1] twice within docker container based on
> flink-1.11.1 scala_2.11-java8-debian image [2], from the results of state
> benchmarks, I cannot see obvious performance changes whether to use
> jemalloc as memory allocator:
>
> RocksDB keyed state backend ops/ms
> glibc   jemalloc        comparison
> ListStateBenchmark.listAdd              537.613 549.926 2.29%
> ListStateBenchmark.listAddAll           301.764 295.51  -2.07%
> ListStateBenchmark.listAppend           521.32  522.614 0.25%
> ListStateBenchmark.listGet              139.19  141.321 1.53%
> ListStateBenchmark.listGetAndIterate    139.685 141.871 1.56%
> ListStateBenchmark.listUpdate           534.785 559.509 4.62%
> MapStateBenchmark.mapAdd                469.748 475.241 1.17%
> MapStateBenchmark.mapContains           51.481  52.188  1.37%
> MapStateBenchmark.mapEntries            352.439 357.951 1.56%
> MapStateBenchmark.mapGet                51.903  52.065  0.31%
> MapStateBenchmark.mapIsEmpty            47.38   48.16   1.65%
> MapStateBenchmark.mapIterator           351.41  357.412 1.71%
> MapStateBenchmark.mapKeys               361.339 359.773 -0.43%
> MapStateBenchmark.mapPutAll             117.067 111.842 -4.46%
> MapStateBenchmark.mapRemove             497.361 499.771 0.48%
> MapStateBenchmark.mapUpdate             464.865 463.501 -0.29%
> MapStateBenchmark.mapValues             350.942 358.64  2.19%
> ValueStateBenchmark.valueAdd            475.55  462.627 -2.72%
> ValueStateBenchmark.valueGet            713.389 729.126 2.21%
> ValueStateBenchmark.valueUpdate         476.373 482.183 1.22%
>
> Heap keyed state backend ops/ms
> glibc   jemalloc        comparison
> ListStateBenchmark.listAdd              6711    6614.719        -1.43%
> ListStateBenchmark.listAddAll           700.429 713.487 1.86%
> ListStateBenchmark.listAppend           2841.068        2848.416
> 0.26%
> ListStateBenchmark.listGet              2863.704        2835.862
> -0.97%
> ListStateBenchmark.listGetAndIterate    2790.001        2787.145
> -0.10%
> ListStateBenchmark.listUpdate           2802.287        2802.626
> 0.01%
> MapStateBenchmark.mapAdd                1939.755        1950.759
> 0.57%
> MapStateBenchmark.mapContains           1914.49 1943.529        1.52%
> MapStateBenchmark.mapEntries            11836.215       11836.673
>  0.00%
> MapStateBenchmark.mapGet                1753.817        1756.643
> 0.16%
> MapStateBenchmark.mapIsEmpty            2980.299        2960.752
> -0.66%
> MapStateBenchmark.mapIterator           11151.177       11123.037
>  -0.25%
> MapStateBenchmark.mapKeys               12956.381       12778.626
>  -1.37%
> MapStateBenchmark.mapPutAll             1253.269        1247.15 -0.49%
> MapStateBenchmark.mapRemove             2594.152        2575.233
> -0.73%
> MapStateBenchmark.mapUpdate             1865.745        1880.573
> 0.79%
> MapStateBenchmark.mapValues             12546.359       12473.223
>  -0.58%
> ValueStateBenchmark.valueAdd            3947.228        3932.003
> -0.39%
> ValueStateBenchmark.valueGet            3887.003        3863.13 -0.61%
> ValueStateBenchmark.valueUpdate         3978.979        3973.183
> -0.15%
>
>
> [1] https://github.com/apache/flink-benchmarks
> [2]
> https://github.com/apache/flink-docker/tree/master/1.11/scala_2.11-java8-debian
>
>
> Best
> Yun Tang
>
> ________________________________
> From: Yun Tang <[hidden email]>
> Sent: Wednesday, November 4, 2020 15:41
> To: [hidden email] <[hidden email]>
> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator
> for debian based Flink docker image
>
> Hi @ Billy
>
> I found there existed many benchmark case existed in the two repos, which
> benchmark case did you run?
>
>
> Best
> Yun Tang
> ________________________________
> From: Xie Billy <[hidden email]>
> Sent: Tuesday, November 3, 2020 22:08
> To: [hidden email] <[hidden email]>
> Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator
> for debian based Flink docker image
>
> Hi guys:
>      refer:
>
>
> https://stackoverflow.com/questions/13027475/cpu-and-memory-usage-of-jemalloc-as-compared-to-glibc-malloc
>      code: https://github.com/jemalloc/jemalloc-experiments
>               https://code.woboq.org/userspace/glibc/benchtests/
>
>
>
>
> Best Regards!
> Billy xie(谢志民)
>
>
> On Fri, Oct 30, 2020 at 4:27 PM Yun Tang <[hidden email]> wrote:
>
> > Hi
> >
> > > Do you see a noticeable performance difference between the two?
> > @ Stephan Ewen , as we already use jemalloc as default memory allocator
> in
> > production, we do not have much experience to compare performace between
> > glibc and jemalloc. And I did not take a look at the performance
> difference
> > when I debug docker OOM. I'll have a try to run benchmark on docker with
> > different allocators when I have time these days.
> >
> > @wang gang, yes, that is what I also observed when I pmap the memory when
> > using glibc, many memory segments with 64MB size.
> >
> > @ Billy Xie, what kind of test case did you use? In my point of view,
> > compared to who would use more memory in some cases, we should care more
> > about who would behave under the max limit for most cases.
> >
> > Best
> > Yun Tang
> > ________________________________
> > From: Xie Billy <[hidden email]>
> > Sent: Friday, October 30, 2020 8:46
> > To: [hidden email] <[hidden email]>
> > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator
> > for debian based Flink docker image
> >
> > After long time  running test application(>96 hours) , glibc used less
> > memory than jemalloc , almost half of jemalloc. Can refer this test case.
> >
> > Best Regards!
> > Billy xie(谢志民)
> >
> >
> > On Fri, Oct 30, 2020 at 12:38 AM 王刚 <[hidden email]>
> wrote:
> >
> > > we  also met glic memory leak .it appeared many 64 MB memory segment
> when
> > > using pmap command . i am +1 on setting jemalloc as default
> > >
> > > 发送自autohome
> > > ________________________________
> > > 发件人: Yu Li <[hidden email]>
> > > 发送时间: 2020-10-30 00:15:12
> > > 收件人: dev <[hidden email]>
> > > 抄送: [hidden email] <[hidden email]>
> > > 主题: Re: [DISCUSS][docker] Adopt Jemalloc as default memory allocator
> for
> > > debian based Flink docker image
> > >
> > > Thanks for sharing more thoughts guys.
> > >
> > > My experience with malloc libraries are also limited, and as mentioned
> in
> > > my previous email [1], my major concern comes from some online
> > information
> > > [2] which indicates in some cases jemalloc will consume as much as
> twice
> > > the memory than glibc, which is a huge cost from my point of view.
> > However,
> > > one may also argue that this post was written in 2015 and things might
> > have
> > > been improved a lot ever since.
> > >
> > > Nevertheless, as Till and Yun pointed out, I could also see the
> benefits
> > of
> > > setting jemalloc as default, so I don't have strong objections here.
> But
> > > this actually is a change of the default memory allocator of our docker
> > > image (previously glibc is the only choice) which definitely worth a
> big
> > > release note.
> > >
> > > Best Regards,
> > > Yu
> > >
> > > [1]
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643p45692.html
> > > [2] https://stackoverflow.com/a/33993215
> > >
> > >
> > > On Thu, 29 Oct 2020 at 20:55, Stephan Ewen <[hidden email]<mailto:
> > > [hidden email]>> wrote:
> > >
> > > > I think Till has a good point. The default should work well, because
> > very
> > > > few users will ever change it. Even on OOMs.
> > > > So I am slightly in favor of going with jemalloc.
> > > >
> > > > @Yun Tang - Do you see a noticeable performance difference between
> the
> > > two?
> > > >
> > > > On Thu, Oct 29, 2020 at 12:32 PM Yun Tang <[hidden email]<mailto:
> > > [hidden email]>> wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I think Till's view deserves wider discussion.
> > > > > I want to give my two cents when I debug with Nico on his reported
> > > > RocksDB
> > > > > OOM problem.
> > > > > Jemalloc has the mechanism to profile memory allocation [1] which
> is
> > > > > widely used to analysis memory leak.
> > > > > Once we set jemalloc as default memory allocator, the frequency of
> > OOM
> > > > > behavior decreases obviously.
> > > > >
> > > > > Considering the OOM killed problem in k8s, change default memory
> > > > allocator
> > > > > as jemalloc could be something beneficial.
> > > > >
> > > > > [1]
> > > https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > >
> > > > > ________________________________
> > > > > From: Till Rohrmann <[hidden email]<mailto:
> > [hidden email]
> > > >>
> > > > > Sent: Thursday, October 29, 2020 18:34
> > > > > To: dev <[hidden email]<mailto:[hidden email]>>
> > > > > Cc: Yun Tang <[hidden email]<mailto:[hidden email]>>
> > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as default memory
> > > allocator
> > > > > for debian based Flink docker image
> > > > >
> > > > > Hi Yu,
> > > > >
> > > > > I see your point why you are in favour of continuing using glibc.
> > > > >
> > > > > I honestly don't have a lot of experience with malloc libraries so
> I
> > > can
> > > > > only argue from a general perspective: We know that glibc has some
> > > > problems
> > > > > wrt to memory fragmentation which can cause processes to exceed its
> > > > memory
> > > > > limit. jemalloc seems to address this problem by avoiding memory
> > > > > fragmentation. However, it might be the case that jemalloc
> introduces
> > > > other
> > > > > problems. In order to figure this out we would need to collect more
> > > data
> > > > > points/experience. So in the end we have to weigh glibc memory
> > > > > fragmentation against unknown problems in jemalloc.
> > > > >
> > > > > Given that choosing the malloc library will be quite an expert
> > > setting, I
> > > > > don't expect many Flink users to use it in case they see OOMs.
> > Hence, I
> > > > > would be slightly in favour of choosing jemalloc as the default
> > because
> > > > it
> > > > > would allow us to gather feedback from users and solve the existing
> > > > > problem. In the best case, user's won't run into new problems. If
> > they
> > > > do,
> > > > > then we will have to re-evaluate this decision and might have to
> > switch
> > > > the
> > > > > malloc library back. W/o setting jemalloc as the default, I fear
> that
> > > we
> > > > > will never gather enough feedback to make jemalloc confidently the
> > > > default.
> > > > >
> > > > > What one could argue, though, is that making jemalloc the default
> so
> > > > > shortly before the feature freeze limits the timespan we can
> observe
> > > how
> > > > it
> > > > > behaves before the actual release. Doing such a change at the
> > beginning
> > > > of
> > > > > a release cycle, would allow us to gain more confidence.
> > > > >
> > > > > In any way, I believe that we should add a big release note and
> > > document
> > > > > what users should do in case they see memory issues (OOM kills,
> > slower
> > > > > performance, etc.).
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Tue, Oct 20, 2020 at 5:39 PM Yu Li <[hidden email]<mailto:
> > > [hidden email]>> wrote:
> > > > >
> > > > > > True, thanks for the reminder Till!
> > > > > >
> > > > > > I suggest using glibc malloc as default. On one hand this follows
> > our
> > > > old
> > > > > > behavior (only with glibc malloc support in the image), on the
> > other
> > > > > hand I
> > > > > > believe glibc isn't using jemalloc as its default memory
> allocator
> > > for
> > > > > some
> > > > > > reason.
> > > > > >
> > > > > > Please let me know your thoughts. Thanks.
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Tue, 20 Oct 2020 at 21:45, Till Rohrmann <
> [hidden email]
> > > <mailto:[hidden email]>>
> > > > > wrote:
> > > > > >
> > > > > > > The only question left would be what will be the default value?
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Tue, Oct 20, 2020 at 10:16 AM Yu Li <[hidden email]
> <mailto:
> > > [hidden email]>> wrote:
> > > > > > >
> > > > > > > > I'm also +1 on making it configurable in the same Docker
> image.
> > > > > > > >
> > > > > > > > It seems we have reached consensus and there are already
> enough
> > > +1s
> > > > > to
> > > > > > > move
> > > > > > > > forward, and suggest @Yun to conclude the discussion directly
> > if
> > > > > there
> > > > > > > are
> > > > > > > > no objections.
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Yu
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, 16 Oct 2020 at 23:16, Till Rohrmann <
> > > [hidden email]<mailto:[hidden email]>>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 for making it configurable in the same Docker image.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Till
> > > > > > > > >
> > > > > > > > > On Fri, Oct 16, 2020 at 12:56 PM Chesnay Schepler <
> > > > > > [hidden email]<mailto:<br></a>> > > [hidden email]>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > If it is possible to support both allocators in a single
> > > image
> > > > > then
> > > > > > > we
> > > > > > > > > > should definitely go with that option.
> > > > > > > > > >
> > > > > > > > > > On 10/16/2020 12:21 PM, Yun Tang wrote:
> > > > > > > > > > > Thanks for Yang's suggestion. I think this could be a
> > > better
> > > > > > > choice.
> > > > > > > > > > > We could install jemalloc and only enable it in
> > LD_PRELOAD
> > > > when
> > > > > > > user
> > > > > > > > > > pass specific configuration for docker-entrypoint.sh.
> > > > > > > > > > > By doing so, we could avoid to create another docker
> > image
> > > > tags
> > > > > > and
> > > > > > > > > also
> > > > > > > > > > offer ability to reduce memory fragmentation problem.
> > > > > > > > > > >
> > > > > > > > > > > Does anyone else have other ideas?
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > Yun Tang
> > > > > > > > > > > ________________________________
> > > > > > > > > > > From: Yang Wang <[hidden email]<mailto:
> > > [hidden email]>>
> > > > > > > > > > > Sent: Thursday, October 15, 2020 14:59
> > > > > > > > > > > To: dev <[hidden email]<mailto:
> > [hidden email]
> > > >>
> > > > > > > > > > > Subject: Re: [DISCUSS][docker] Adopt Jemalloc as
> default
> > > > memory
> > > > > > > > > > allocator for debian based Flink docker image
> > > > > > > > > > >
> > > > > > > > > > > Thanks Yun Tang for starting this discussion.
> > > > > > > > > > >
> > > > > > > > > > > I think this is very important when deploy Flink with
> > > > container
> > > > > > > > > > environment
> > > > > > > > > > > in production. I just
> > > > > > > > > > > have quick question. Could we have both memory
> > > allocator(e.g.
> > > > > > > glibc,
> > > > > > > > > > > jemalloc) in the Flink
> > > > > > > > > > > official image and enable a specific one by setting
> ENV?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yang
> > > > > > > > > > >
> > > > > > > > > > > Yu Li <[hidden email]<mailto:[hidden email]>>
> > > 于2020年10月14日周三 下午12:23写道：
> > > > > > > > > > >
> > > > > > > > > > >> Thanks for debugging and resolving the issue and
> driving
> > > the
> > > > > > > > > discussion
> > > > > > > > > > >> Yun!
> > > > > > > > > > >>
> > > > > > > > > > >> For the given solutions, I prefer option 1 (supply
> > another
> > > > > > > > Dockerfile
> > > > > > > > > > using
> > > > > > > > > > >> jemalloc as default memory allocator) because of the
> > below
> > > > > > > reasons:
> > > > > > > > > > >>
> > > > > > > > > > >> 1. It's hard to say jemalloc is always better than
> > > ptmalloc
> > > > > > (glibc
> > > > > > > > > > malloc),
> > > > > > > > > > >> or else glibc should have already adopted it as the
> > > default
> > > > > > memory
> > > > > > > > > > >> allocator. And as indicated here [1], in some cases
> > > jemalloc
> > > > > > will
> > > > > > > > > > >> consume as much as twice the memory than glibc
> > > > > > > > > > >>
> > > > > > > > > > >> 2. All existing Flink docker images use glibc, if we
> > > change
> > > > > the
> > > > > > > > > default
> > > > > > > > > > >> memory allocator to jemalloc and only supply one
> series
> > of
> > > > > > images,
> > > > > > > > we
> > > > > > > > > > will
> > > > > > > > > > >> leave those having better performance with glibc no
> > other
> > > > > > choices
> > > > > > > > but
> > > > > > > > > > >> staying with old images. In another word, there's a
> risk
> > > of
> > > > > > > > > introducing
> > > > > > > > > > new
> > > > > > > > > > >> problems while fixing an existing one if choosing
> > > option-2.
> > > > > > > > > > >>
> > > > > > > > > > >> And there is a third option considering the efforts of
> > > > > > maintaining
> > > > > > > > > more
> > > > > > > > > > >> images if the memory leak issue is not widely
> observed,
> > > that
> > > > > we
> > > > > > > > could
> > > > > > > > > > >> document the steps of building Dockerfile with
> jemalloc
> > as
> > > > > > default
> > > > > > > > > > >> allocator so users could build it when needed, which
> > > leaves
> > > > > the
> > > > > > > > burden
> > > > > > > > > > to
> > > > > > > > > > >> our users so for me it's not the best option.
> > > > > > > > > > >>
> > > > > > > > > > >> Best Regards,
> > > > > > > > > > >> Yu
> > > > > > > > > > >>
> > > > > > > > > > >> [1] https://stackoverflow.com/a/33993215
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, 13 Oct 2020 at 15:34, Yun Tang <
> > [hidden email]
> > > <mailto:[hidden email]>>
> > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi all
> > > > > > > > > > >>>
> > > > > > > > > > >>> Users report they meet serious memory leak when
> > > submitting
> > > > > jobs
> > > > > > > > > > >>> continously in session mode within k8s (please refer
> to
> > > > > > > > > FLINK-18712[1]
> > > > > > > > > > ),
> > > > > > > > > > >>> and I also reproduce this to find this is caused by
> > > memory
> > > > > > > > > > fragmentation
> > > > > > > > > > >> of
> > > > > > > > > > >>> glibc [2][3] and provide solutions to fix this:
> > > > > > > > > > >>>
> > > > > > > > > > >>>    *   Quick but not very clean solution to limit the
> > > > memory
> > > > > > pool
> > > > > > > > of
> > > > > > > > > > >> glibc,
> > > > > > > > > > >>> limit MALLOC_ARENA_MAX to 2
> > > > > > > > > > >>>
> > > > > > > > > > >>>    *   More general solution by rebuilding the image
> to
> > > > > install
> > > > > > > > > > >>> libjemalloc-dev and add the libjemalloc.so it to
> > > LD_PRELOAD
> > > > > > > > > > >>>
> > > > > > > > > > >>> The reporter adopted the 2nd solution to fix this
> issue
> > > > > > > eventually.
> > > > > > > > > > Thus,
> > > > > > > > > > >>> I begin to think whether we should change our
> > Dockerfile
> > > to
> > > > > > adopt
> > > > > > > > > > >> jemalloc
> > > > > > > > > > >>> as default memory allocator [4].
> > > > > > > > > > >>>  From my point of view, we have two choices:
> > > > > > > > > > >>>
> > > > > > > > > > >>>    1.  Introduce another Dockerfile using jemalloc as
> > > > default
> > > > > > > > memory
> > > > > > > > > > >>> allocator, which means Flink needs another two new
> > image
> > > > tags
> > > > > > to
> > > > > > > > > build
> > > > > > > > > > >>> docker with jemalloc while default docker still use
> > > glibc.
> > > > > > > > > > >>>    2.  Set the default memory allocator as jemalloc
> in
> > > our
> > > > > > > existing
> > > > > > > > > > >>> Dockerfiles, which means Flink offer docker image
> with
> > > > > jemalloc
> > > > > > > by
> > > > > > > > > > >> default.
> > > > > > > > > > >>> I prefer the 2nd option as our company already use
> > > jemalloc
> > > > > as
> > > > > > > > > default
> > > > > > > > > > >>> memory allocator for JDK at our production
> environment
> > > due
> > > > to
> > > > > > > > > messages
> > > > > > > > > > >> from
> > > > > > > > > > >>> os team warning of glibc's memory fragmentation.
> > > > > > > > > > >>> Moreover, I found several open source projects
> adopting
> > > > > > jemalloc
> > > > > > > as
> > > > > > > > > > >>> default memory allocator within their images to
> resolve
> > > > > memory
> > > > > > > > > > >>> fragmentation problem, e.g fluent [5], home-assistant
> > > [6].
> > > > > > > > > > >>>
> > > > > > > > > > >>> What do you guys think of this issue?
> > > > > > > > > > >>>
> > > > > > > > > > >>> [1]
> https://issues.apache.org/jira/browse/FLINK-18712
> > > > > > > > > > >>> [2]
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.gnu.org/software/libc/manual/html_mono/libc.html#Freeing-after-Malloc
> > > > > > > > > > >>> [3]
> > > https://sourceware.org/bugzilla/show_bug.cgi?id=15321
> > > > > > > > > > >>> [4]
> https://issues.apache.org/jira/browse/FLINK-19125
> > > > > > > > > > >>> [5]
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.fluentbit.io/manual/v/1.0/installation/docker#why-there-is-no-fluent-bit-docker-image-based-on-alpine-linux
> > > > > > > > > > >>> [6]
> https://github.com/home-assistant/core/pull/33237
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> Best
> > > > > > > > > > >>> Yun Tang
> > > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>