Re: Multiple Taskmanagers per node for standalone cluster

[Xintong Song]

I don't think using zookeeper should cause any problem on starting multiple
TMs per node.

For standalone cluster, having one TM per node is usually the easiest way.
It is easy to config (simply config the TM resource to the available
resource of the node), and it is more efficient (less TM framework
overhead, no isolation between JVMs of different TMs). But that doesn't
mean Flink can not have multiple TMs per node, and given your requirements
of job isolation, I think you should do that. You just need to carefully
config the TM number and resources to make sure they do not exceed the
node's available resource.

I don't have much experience running standalone clusters with multiple TMs
per node. We mainly run Flink on Yarn in our production, with multiple TMs
(Yarn containers) per node (Yarn NM), and it works fine for us.

Thank you~

Xintong Song



On Mon, Oct 7, 2019 at 9:05 PM Ethan Li <[hidden email]> wrote:

> Thank you very much Xintong for the information.
>
> I am currently using zookeeper. I guess starting another task manager will
> just work? But I am not sure if multiple TaskManagers per node is good idea
> to use Flink. Do you have any experience on this? Thanks!
>
> Best,
> Ethan
>
>
> On Oct 5, 2019, at 3:43 AM, Xintong Song <[hidden email]> wrote:
>
> For having multiple task managers on the same host, you can put multiple
> duplicated lines with the target host in '<your-flink-home>/conf/slaves'.
> The task managers on the same host will share the same config files on the
> host.
>
> Thank you~
> Xintong Song
>
>
>
> On Sat, Oct 5, 2019 at 5:02 AM Ethan Li <[hidden email]> wrote:
>
>> Hello,
>>
>> Does/did anyone try to set up a standalone cluster with multiple
>> TaskManagers per node? We are working on moving to flink-on-yarn solution.
>> But before that happens,  I am thinking about the following setup to  get
>> jobs isolated from each other
>>
>> 1) multiple taskmanager per host
>> 2) 1 taskSlot per TaskManager
>>
>>
>> Currently we have 1 TaskManger per node and many taskSlot per TM, tasks
>> from different job will be scheduled into one JVM process and it’s
>> basically impossible to debug. One bad job will kill the whole cluster.
>>
>> Could you please share if you have any experience on this and what’re the
>> problems that might have?
>>
>> Thank you very much. Really appreciate it.
>>
>> Best,
>> Ethan
>
>
>

[Ethan Li]

Thank you very much for the help, Xintong.

Best,
Ethan

> On Oct 7, 2019, at 9:18 PM, Xintong Song <[hidden email]> wrote:
>
> I don't think using zookeeper should cause any problem on starting multiple TMs per node.
>
> For standalone cluster, having one TM per node is usually the easiest way. It is easy to config (simply config the TM resource to the available resource of the node), and it is more efficient (less TM framework overhead, no isolation between JVMs of different TMs). But that doesn't mean Flink can not have multiple TMs per node, and given your requirements of job isolation, I think you should do that. You just need to carefully config the TM number and resources to make sure they do not exceed the node's available resource.
>
> I don't have much experience running standalone clusters with multiple TMs per node. We mainly run Flink on Yarn in our production, with multiple TMs (Yarn containers) per node (Yarn NM), and it works fine for us.
>
> Thank you~
> Xintong Song
>
>
> On Mon, Oct 7, 2019 at 9:05 PM Ethan Li <[hidden email] <mailto:[hidden email]>> wrote:
> Thank you very much Xintong for the information.
>
> I am currently using zookeeper. I guess starting another task manager will just work? But I am not sure if multiple TaskManagers per node is good idea to use Flink. Do you have any experience on this? Thanks!
>
> Best,
> Ethan
>
>
>> On Oct 5, 2019, at 3:43 AM, Xintong Song <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> For having multiple task managers on the same host, you can put multiple duplicated lines with the target host in '<your-flink-home>/conf/slaves'. The task managers on the same host will share the same config files on the host.
>>
>> Thank you~
>> Xintong Song
>>
>>
>> On Sat, Oct 5, 2019 at 5:02 AM Ethan Li <[hidden email] <mailto:[hidden email]>> wrote:
>> Hello,
>>
>> Does/did anyone try to set up a standalone cluster with multiple TaskManagers per node? We are working on moving to flink-on-yarn solution. But before that happens,  I am thinking about the following setup to  get jobs isolated from each other
>>
>> 1) multiple taskmanager per host
>> 2) 1 taskSlot per TaskManager
>>
>>
>> Currently we have 1 TaskManger per node and many taskSlot per TM, tasks from different job will be scheduled into one JVM process and it’s basically impossible to debug. One bad job will kill the whole cluster.
>>
>> Could you please share if you have any experience on this and what’re the problems that might have?
>>
>> Thank you very much. Really appreciate it.
>>
>> Best,
>> Ethan
>