Flink Cluster Load Distribution Question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink Cluster Load Distribution Question

amir bahmanyari
Hi Colleagues,Just joined this forum.I have done everything possible to get a 4 nodes Flink cluster to work peoperly & run a Beam app.It always generates system-output logs (*.out) in only one node. Its sooooooooo slow for 4 nodes being there.Seems like the load is not distributed amongst all 4 nodes but only one node. Most of the time the one where JM runs.I run/tested it in a single node, and it took even faster to run the same load.Not sure whats not being configured right.1- why am I getting SystemOut .out log in only one server? All nodes get their TaskManager log files updated thu.2- why dont I see load being distributed amongst all 4 nodes, but only one all the times.3- Why does the Dashboard show a 0 (zero) for Send/Receive numbers per all Task Managers.
The Dashboard shows all the right stuff. Top shows not much of resources being stressed on any of the nodes.I can share its contents if it helps diagnosing the issue.Thanks + I appreciate your valuable time, response & help.Amir-
Reply | Threaded
Open this post in threaded view
|

Re: Flink Cluster Load Distribution Question

Robert Metzger
Hi Amir,

I would recommend to post such questions to the user@flink mailing list in
the future. This list is meant for development-related topics.

I think we need more details to understand why your application is not
running properly. Can you quickly describe what your topology is doing?
Are you setting the parallelism to a value >= 1 ?

Regards,
Robert


On Tue, Sep 13, 2016 at 6:35 AM, amir bahmanyari <
[hidden email]> wrote:

> Hi Colleagues,Just joined this forum.I have done everything possible to
> get a 4 nodes Flink cluster to work peoperly & run a Beam app.It always
> generates system-output logs (*.out) in only one node. Its sooooooooo slow
> for 4 nodes being there.Seems like the load is not distributed amongst all
> 4 nodes but only one node. Most of the time the one where JM runs.I
> run/tested it in a single node, and it took even faster to run the same
> load.Not sure whats not being configured right.1- why am I getting
> SystemOut .out log in only one server? All nodes get their TaskManager log
> files updated thu.2- why dont I see load being distributed amongst all 4
> nodes, but only one all the times.3- Why does the Dashboard show a 0 (zero)
> for Send/Receive numbers per all Task Managers.
> The Dashboard shows all the right stuff. Top shows not much of resources
> being stressed on any of the nodes.I can share its contents if it helps
> diagnosing the issue.Thanks + I appreciate your valuable time, response &
> help.Amir-
Reply | Threaded
Open this post in threaded view
|

Re: Flink Cluster Load Distribution Question

amir bahmanyari
Hi Robert,I forwarded my Qs to the users forum...No response ...Is there anyway I can get some answers to my questions pls?I really really appreciate your help...CheersAmir-

      From: Robert Metzger <[hidden email]>
 To: "[hidden email]" <[hidden email]>; amir bahmanyari <[hidden email]>
 Sent: Tuesday, September 13, 2016 1:15 AM
 Subject: Re: Flink Cluster Load Distribution Question
   
Hi Amir,

I would recommend to post such questions to the user@flink mailing list in
the future. This list is meant for development-related topics.

I think we need more details to understand why your application is not
running properly. Can you quickly describe what your topology is doing?
Are you setting the parallelism to a value >= 1 ?

Regards,
Robert


On Tue, Sep 13, 2016 at 6:35 AM, amir bahmanyari <
[hidden email]> wrote:

> Hi Colleagues,Just joined this forum.I have done everything possible to
> get a 4 nodes Flink cluster to work peoperly & run a Beam app.It always
> generates system-output logs (*.out) in only one node. Its sooooooooo slow
> for 4 nodes being there.Seems like the load is not distributed amongst all
> 4 nodes but only one node. Most of the time the one where JM runs.I
> run/tested it in a single node, and it took even faster to run the same
> load.Not sure whats not being configured right.1- why am I getting
> SystemOut .out log in only one server? All nodes get their TaskManager log
> files updated thu.2- why dont I see load being distributed amongst all 4
> nodes, but only one all the times.3- Why does the Dashboard show a 0 (zero)
> for Send/Receive numbers per all Task Managers.
> The Dashboard shows all the right stuff. Top shows not much of resources
> being stressed on any of the nodes.I can share its contents if it helps
> diagnosing the issue.Thanks + I appreciate your valuable time, response &
> help.Amir-