Flink memory consumption

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink memory consumption

Ramya Ramamurthy
Hi,

I have Flink 1.7 running on our production. I can see that the Memory used by the Job Managers are pretty high. Below is a snapshot of our pods running the JM. 
I had to commit 3GB memory, with a memory limit of 5GB. I have one cluster per job, to make it scalable based on our traffic rate and the benchmark per pod. So this cluster holds a job with 5 parallelism. 
I just wanted to know how to understand if this memory consumption is acceptable. I was assuming JM's might not need this much memory. Please advise.


Reply | Threaded
Open this post in threaded view
|

Re: Flink memory consumption

Xintong Song
Hi Ramya,

The attached image cannot be displayed.

It does not sound normal to me that your job manager with only one job of 5
parallelisms needs 3GB. Have you enabled many per-record metrics? And have
you tried with the latest Flink versions (1.11 or 1.10)?

Thank you~

Xintong Song



On Fri, Jul 10, 2020 at 2:32 PM Ramya Ramamurthy <[hidden email]> wrote:

> Hi,
>
> I have Flink 1.7 running on our production. I can see that the Memory used
> by the Job Managers are pretty high. Below is a snapshot of our pods
> running the JM.
> I had to commit 3GB memory, with a memory limit of 5GB. I have one cluster
> per job, to make it scalable based on our traffic rate and the benchmark
> per pod. So this cluster holds a job with 5 parallelism.
> I just wanted to know how to understand if this memory consumption is
> acceptable. I was assuming JM's might not need this much memory. Please
> advise.
>
> [image: image.png]
>