Re: How jobmanager and task manager communicates with each other ?

[sidhant gupta]

++[hidden email]

On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <[hidden email]> wrote:

> Hi User
>
> How jobmanager and task manager communicates with each other ? How to set
> connection between jobmanager and task manager running in different/same
> ec2 instance ? Is it http or tcp ? How the service discovery works ?
>
> Thanks
> Sidhant Gupta
>

[Till Rohrmann]

Hi Sidhant,

the cluster components use tcp to communicate with each other. If you are
not using Flink's HA services, then the TaskManager nodes need to be
configured with the JobManager's address to connect to them. If you are
using HA services, then the service discovery happens through the HA
services. One requirement for Flink to work is that the different cluster
nodes on which a Flink process is started can communicate with each other.

Cheers,
Till

On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <[hidden email]> wrote:

> ++[hidden email]
>
> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <[hidden email]> wrote:
>
> > Hi User
> >
> > How jobmanager and task manager communicates with each other ? How to set
> > connection between jobmanager and task manager running in different/same
> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
> >
> > Thanks
> > Sidhant Gupta
> >
>

[sidhant gupta]

Hi Till,

Thanks for the reply.

(1) If we are not using Flink's HA services then how we can dynamically
configure task manager nodes to connect to job manager? Any suggestions or
best practices?

(2) Which and how flink's HA service can be used for the service discovery
of job manager ?

Regards
Sidhant Gupta


On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann <[hidden email]> wrote:

> Hi Sidhant,
>
> the cluster components use tcp to communicate with each other. If you are
> not using Flink's HA services, then the TaskManager nodes need to be
> configured with the JobManager's address to connect to them. If you are
> using HA services, then the service discovery happens through the HA
> services. One requirement for Flink to work is that the different cluster
> nodes on which a Flink process is started can communicate with each other.
>
> Cheers,
> Till
>
> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <[hidden email]> wrote:
>
>> ++[hidden email]
>>
>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <[hidden email]> wrote:
>>
>> > Hi User
>> >
>> > How jobmanager and task manager communicates with each other ? How to
>> set
>> > connection between jobmanager and task manager running in different/same
>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>> >
>> > Thanks
>> > Sidhant Gupta
>> >
>>
>

[Andrey Zagrebin-4]

Hi Sidhant,

(1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?

Not sure what you mean by 'dynamically'.
I think you have to restart the task manager with the new configuration
to connect to another job manager.

(2) Which and how flink's HA service can be used for the service discovery
> of job manager ?

You can check the docs for the zookeeper implementation of the HA in Flink
[1]

Best,
Andrey

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/jobmanager_high_availability.html

On Tue, Aug 25, 2020 at 5:45 PM sidhant gupta <[hidden email]> wrote:

> Hi Till,
>
> Thanks for the reply.
>
> (1) If we are not using Flink's HA services then how we can dynamically
> configure task manager nodes to connect to job manager? Any suggestions or
> best practices?
>
> (2) Which and how flink's HA service can be used for the service discovery
> of job manager ?
>
> Regards
> Sidhant Gupta
>
>
> On Tue, Aug 25, 2020, 11:51 AM Till Rohrmann <[hidden email]> wrote:
>
>> Hi Sidhant,
>>
>> the cluster components use tcp to communicate with each other. If you are
>> not using Flink's HA services, then the TaskManager nodes need to be
>> configured with the JobManager's address to connect to them. If you are
>> using HA services, then the service discovery happens through the HA
>> services. One requirement for Flink to work is that the different cluster
>> nodes on which a Flink process is started can communicate with each other.
>>
>> Cheers,
>> Till
>>
>> On Mon, Aug 24, 2020 at 6:26 PM sidhant gupta <[hidden email]>
>> wrote:
>>
>>> ++[hidden email]
>>>
>>> On Mon, Aug 24, 2020, 7:31 PM sidhant gupta <[hidden email]> wrote:
>>>
>>> > Hi User
>>> >
>>> > How jobmanager and task manager communicates with each other ? How to
>>> set
>>> > connection between jobmanager and task manager running in
>>> different/same
>>> > ec2 instance ? Is it http or tcp ? How the service discovery works ?
>>> >
>>> > Thanks
>>> > Sidhant Gupta
>>> >
>>>
>>