How to set jobmanager.rpc.address in TaskManger node in HA cluster

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to set jobmanager.rpc.address in TaskManger node in HA cluster

Mu Kong
Hi all,

I'm trying setup an HA Flink cluster with 3 job managers and 3 task
managers.
After executing start-cluster.sh, 3 job managers are normally started.
However, the task managers are still down due to an exception showed below:

org.apache.flink.util.ConfigurationException: Config parameter 'Key:
'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing
(hostname/address of JobManager to connect to).


I managed to setup a non-HA cluster before and I know I should put the Job
manager's address here, but what should I put here in HA cluster since
there are three job managers here.


Besides, according to the original config file here:

https://github.com/apache/flink/blob/master/flink-dist/src/main/resources/flink-conf.yaml#L28

this should be taken care of automatically.

I have already put masters under /conf folder with all the job managers'
addresses/ports in it. So the exception shouldn't have shown in the first
place.


Is there anything else I have missed?


Thanks in advance.

Mu
Reply | Threaded
Open this post in threaded view
|

Re: How to set jobmanager.rpc.address in TaskManger node in HA cluster

Mu Kong
Sorry. I didn't put high-availability: zookeeper in taskmangers'
flink-config.yml.
After I fixed this, everything went well.

On Fri, Jul 7, 2017 at 11:08 AM, Mu Kong <[hidden email]> wrote:

> Hi all,
>
> I'm trying setup an HA Flink cluster with 3 job managers and 3 task
> managers.
> After executing start-cluster.sh, 3 job managers are normally started.
> However, the task managers are still down due to an exception showed below:
>
> org.apache.flink.util.ConfigurationException: Config parameter 'Key:
> 'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing
> (hostname/address of JobManager to connect to).
>
>
> I managed to setup a non-HA cluster before and I know I should put the Job
> manager's address here, but what should I put here in HA cluster since
> there are three job managers here.
>
>
> Besides, according to the original config file here:
>
> https://github.com/apache/flink/blob/master/flink-dist/
> src/main/resources/flink-conf.yaml#L28
>
> this should be taken care of automatically.
>
> I have already put masters under /conf folder with all the job managers'
> addresses/ports in it. So the exception shouldn't have shown in the first
> place.
>
>
> Is there anything else I have missed?
>
>
> Thanks in advance.
>
> Mu
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to set jobmanager.rpc.address in TaskManger node in HA cluster

Aljoscha Krettek-2
Cool, thanks for letting us know that you figured it out and what it was!

> On 7. Jul 2017, at 05:52, Mu Kong <[hidden email]> wrote:
>
> Sorry. I didn't put high-availability: zookeeper in taskmangers'
> flink-config.yml.
> After I fixed this, everything went well.
>
> On Fri, Jul 7, 2017 at 11:08 AM, Mu Kong <[hidden email]> wrote:
>
>> Hi all,
>>
>> I'm trying setup an HA Flink cluster with 3 job managers and 3 task
>> managers.
>> After executing start-cluster.sh, 3 job managers are normally started.
>> However, the task managers are still down due to an exception showed below:
>>
>> org.apache.flink.util.ConfigurationException: Config parameter 'Key:
>> 'jobmanager.rpc.address' , default: null (deprecated keys: [])' is missing
>> (hostname/address of JobManager to connect to).
>>
>>
>> I managed to setup a non-HA cluster before and I know I should put the Job
>> manager's address here, but what should I put here in HA cluster since
>> there are three job managers here.
>>
>>
>> Besides, according to the original config file here:
>>
>> https://github.com/apache/flink/blob/master/flink-dist/
>> src/main/resources/flink-conf.yaml#L28
>>
>> this should be taken care of automatically.
>>
>> I have already put masters under /conf folder with all the job managers'
>> addresses/ports in it. So the exception shouldn't have shown in the first
>> place.
>>
>>
>> Is there anything else I have missed?
>>
>>
>> Thanks in advance.
>>
>> Mu
>>
>>
>>