[DISCUSS] Re-renaming "Flink Master" back to JobManager

[Aljoscha Krettek-2]

Hi All,

This came to my mind because of the master/slave discussion in [1] and
the larger discussions about inequality/civil rights happening right now
in the world. I think for this reason alone we should use a name that
does not include "master".

We could rename it back to JobManager, which was the name mostly used
before 2019. Since the beginning of Flink, TaskManager was the term used
for the worker component/node and JobManager was the term used for the
orchestrating component/node.

Currently our glossary [2] defines these terms (paraphrased by me):

  - "Flink Master": it's the orchestrating component that consists of
resource manager, dispatcher, and JobManager

  - JobManager: it's the thing that manages a single job and runs as
part of a "Flink Master"

  - TaskManager: it's the worker process

Prior to the introduction of the glossary the definition of JobManager
would have been:

  - It's the orchestrating component that manages execution of jobs and
schedules work on TaskManagers.

Quite some parts in the code and documentation/configuration options
still use that older meaning of JobManager. Newer parts of the
documentation use "Flink Master" instead.

I'm proposing to go back to calling the orchestrating component
JobManager, which would mean that we have to touch up the documentation
to remove mentions of "Flink Master". I'm also proposing not to mention
the internal components such as resource manager and dispatcher in the
glossary because there are transparent to users.

I'm proposing to go back to JobManager instead of an alternative name
also because switching to yet another name would mean many more changes
to code/documentation/peoples minds.

What do you all think?

Best,
Aljoscha


[1] https://issues.apache.org/jira/browse/FLINK-18209
[2]
https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html

[Konstantin Knauf-4]

Hi Aljoscha,

sounds good to me. Let’s also make sure we don’t refer to the JobMaster as
Jobmanager anywhere then (code, config).

I am not sure we can avoid mentioning the Flink ResourceManagers in user
facing docs completely. For JobMaster and Dispatcher this seems doable.

Best,

Konstantin

On Mon 15. Jun 2020 at 12:56, Aljoscha Krettek <[hidden email]> wrote:

> Hi All,
>
> This came to my mind because of the master/slave discussion in [1] and
> the larger discussions about inequality/civil rights happening right now
> in the world. I think for this reason alone we should use a name that
> does not include "master".
>
> We could rename it back to JobManager, which was the name mostly used
> before 2019. Since the beginning of Flink, TaskManager was the term used
> for the worker component/node and JobManager was the term used for the
> orchestrating component/node.
>
> Currently our glossary [2] defines these terms (paraphrased by me):
>
>   - "Flink Master": it's the orchestrating component that consists of
> resource manager, dispatcher, and JobManager
>
>   - JobManager: it's the thing that manages a single job and runs as
> part of a "Flink Master"
>
>   - TaskManager: it's the worker process
>
> Prior to the introduction of the glossary the definition of JobManager
> would have been:
>
>   - It's the orchestrating component that manages execution of jobs and
> schedules work on TaskManagers.
>
> Quite some parts in the code and documentation/configuration options
> still use that older meaning of JobManager. Newer parts of the
> documentation use "Flink Master" instead.
>
> I'm proposing to go back to calling the orchestrating component
> JobManager, which would mean that we have to touch up the documentation
> to remove mentions of "Flink Master". I'm also proposing not to mention
> the internal components such as resource manager and dispatcher in the
> glossary because there are transparent to users.
>
> I'm proposing to go back to JobManager instead of an alternative name
> also because switching to yet another name would mean many more changes
> to code/documentation/peoples minds.
>
> What do you all think?
>
> Best,
> Aljoscha
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-18209
> [2]
>
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
>

--

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

[David Anderson-3]

Aljoscha,

I think this is a step in the right direction.

In some cases it may be difficult to talk concretely about the
differences between different deployment models (e.g., comparing a k8s
per-job cluster to a YARN-based session cluster, which is something I
typically present during training) without giving names to the internal
components. I'm not convinced we can completely avoid mentioning the
JobMaster (and Dispatcher and ResourceManagers) in some (rare) contexts --
but I don't see this as an argument against the proposed change.

David

On Mon, Jun 15, 2020 at 2:32 PM Konstantin Knauf <[hidden email]> wrote:

> Hi Aljoscha,
>
> sounds good to me. Let’s also make sure we don’t refer to the JobMaster as
> Jobmanager anywhere then (code, config).
>
> I am not sure we can avoid mentioning the Flink ResourceManagers in user
> facing docs completely. For JobMaster and Dispatcher this seems doable.
>
> Best,
>
> Konstantin
>
> On Mon 15. Jun 2020 at 12:56, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > Hi All,
> >
> > This came to my mind because of the master/slave discussion in [1] and
> > the larger discussions about inequality/civil rights happening right now
> > in the world. I think for this reason alone we should use a name that
> > does not include "master".
> >
> > We could rename it back to JobManager, which was the name mostly used
> > before 2019. Since the beginning of Flink, TaskManager was the term used
> > for the worker component/node and JobManager was the term used for the
> > orchestrating component/node.
> >
> > Currently our glossary [2] defines these terms (paraphrased by me):
> >
> >   - "Flink Master": it's the orchestrating component that consists of
> > resource manager, dispatcher, and JobManager
> >
> >   - JobManager: it's the thing that manages a single job and runs as
> > part of a "Flink Master"
> >
> >   - TaskManager: it's the worker process
> >
> > Prior to the introduction of the glossary the definition of JobManager
> > would have been:
> >
> >   - It's the orchestrating component that manages execution of jobs and
> > schedules work on TaskManagers.
> >
> > Quite some parts in the code and documentation/configuration options
> > still use that older meaning of JobManager. Newer parts of the
> > documentation use "Flink Master" instead.
> >
> > I'm proposing to go back to calling the orchestrating component
> > JobManager, which would mean that we have to touch up the documentation
> > to remove mentions of "Flink Master". I'm also proposing not to mention
> > the internal components such as resource manager and dispatcher in the
> > glossary because there are transparent to users.
> >
> > I'm proposing to go back to JobManager instead of an alternative name
> > also because switching to yet another name would mean many more changes
> > to code/documentation/peoples minds.
> >
> > What do you all think?
> >
> > Best,
> > Aljoscha
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-18209
> > [2]
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
> >
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

[Robert Metzger]

Thanks a lot for looking into this!

+1 to your proposal

On Wed, Jun 17, 2020 at 10:55 AM David Anderson <[hidden email]>
wrote:

> Aljoscha,
>
> I think this is a step in the right direction.
>
> In some cases it may be difficult to talk concretely about the
> differences between different deployment models (e.g., comparing a k8s
> per-job cluster to a YARN-based session cluster, which is something I
> typically present during training) without giving names to the internal
> components. I'm not convinced we can completely avoid mentioning the
> JobMaster (and Dispatcher and ResourceManagers) in some (rare) contexts --
> but I don't see this as an argument against the proposed change.
>
> David
>
> On Mon, Jun 15, 2020 at 2:32 PM Konstantin Knauf <[hidden email]>
> wrote:
>
> > Hi Aljoscha,
> >
> > sounds good to me. Let’s also make sure we don’t refer to the JobMaster
> as
> > Jobmanager anywhere then (code, config).
> >
> > I am not sure we can avoid mentioning the Flink ResourceManagers in user
> > facing docs completely. For JobMaster and Dispatcher this seems doable.
> >
> > Best,
> >
> > Konstantin
> >
> > On Mon 15. Jun 2020 at 12:56, Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> > > Hi All,
> > >
> > > This came to my mind because of the master/slave discussion in [1] and
> > > the larger discussions about inequality/civil rights happening right
> now
> > > in the world. I think for this reason alone we should use a name that
> > > does not include "master".
> > >
> > > We could rename it back to JobManager, which was the name mostly used
> > > before 2019. Since the beginning of Flink, TaskManager was the term
> used
> > > for the worker component/node and JobManager was the term used for the
> > > orchestrating component/node.
> > >
> > > Currently our glossary [2] defines these terms (paraphrased by me):
> > >
> > >   - "Flink Master": it's the orchestrating component that consists of
> > > resource manager, dispatcher, and JobManager
> > >
> > >   - JobManager: it's the thing that manages a single job and runs as
> > > part of a "Flink Master"
> > >
> > >   - TaskManager: it's the worker process
> > >
> > > Prior to the introduction of the glossary the definition of JobManager
> > > would have been:
> > >
> > >   - It's the orchestrating component that manages execution of jobs and
> > > schedules work on TaskManagers.
> > >
> > > Quite some parts in the code and documentation/configuration options
> > > still use that older meaning of JobManager. Newer parts of the
> > > documentation use "Flink Master" instead.
> > >
> > > I'm proposing to go back to calling the orchestrating component
> > > JobManager, which would mean that we have to touch up the documentation
> > > to remove mentions of "Flink Master". I'm also proposing not to mention
> > > the internal components such as resource manager and dispatcher in the
> > > glossary because there are transparent to users.
> > >
> > > I'm proposing to go back to JobManager instead of an alternative name
> > > also because switching to yet another name would mean many more changes
> > > to code/documentation/peoples minds.
> > >
> > > What do you all think?
> > >
> > > Best,
> > > Aljoscha
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-18209
> > > [2]
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
> > >
> > --
> >
> > Konstantin Knauf
> >
> > https://twitter.com/snntrable
> >
> > https://github.com/knaufk
> >
>

[Marta Paes Moreira]

+1

I've found the term Flink Master a tad bit confusing myself, in the past,
as it's not used consistently throughout the documentation (as you mention).

Thanks for following up on this, Aljoscha!

On Wed, Jun 17, 2020 at 5:16 PM Robert Metzger <[hidden email]> wrote:

> Thanks a lot for looking into this!
>
> +1 to your proposal
>
> On Wed, Jun 17, 2020 at 10:55 AM David Anderson <[hidden email]>
> wrote:
>
> > Aljoscha,
> >
> > I think this is a step in the right direction.
> >
> > In some cases it may be difficult to talk concretely about the
> > differences between different deployment models (e.g., comparing a k8s
> > per-job cluster to a YARN-based session cluster, which is something I
> > typically present during training) without giving names to the internal
> > components. I'm not convinced we can completely avoid mentioning the
> > JobMaster (and Dispatcher and ResourceManagers) in some (rare) contexts
> --
> > but I don't see this as an argument against the proposed change.
> >
> > David
> >
> > On Mon, Jun 15, 2020 at 2:32 PM Konstantin Knauf <[hidden email]>
> > wrote:
> >
> > > Hi Aljoscha,
> > >
> > > sounds good to me. Let’s also make sure we don’t refer to the JobMaster
> > as
> > > Jobmanager anywhere then (code, config).
> > >
> > > I am not sure we can avoid mentioning the Flink ResourceManagers in
> user
> > > facing docs completely. For JobMaster and Dispatcher this seems doable.
> > >
> > > Best,
> > >
> > > Konstantin
> > >
> > > On Mon 15. Jun 2020 at 12:56, Aljoscha Krettek <[hidden email]>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > This came to my mind because of the master/slave discussion in [1]
> and
> > > > the larger discussions about inequality/civil rights happening right
> > now
> > > > in the world. I think for this reason alone we should use a name that
> > > > does not include "master".
> > > >
> > > > We could rename it back to JobManager, which was the name mostly used
> > > > before 2019. Since the beginning of Flink, TaskManager was the term
> > used
> > > > for the worker component/node and JobManager was the term used for
> the
> > > > orchestrating component/node.
> > > >
> > > > Currently our glossary [2] defines these terms (paraphrased by me):
> > > >
> > > >   - "Flink Master": it's the orchestrating component that consists of
> > > > resource manager, dispatcher, and JobManager
> > > >
> > > >   - JobManager: it's the thing that manages a single job and runs as
> > > > part of a "Flink Master"
> > > >
> > > >   - TaskManager: it's the worker process
> > > >
> > > > Prior to the introduction of the glossary the definition of
> JobManager
> > > > would have been:
> > > >
> > > >   - It's the orchestrating component that manages execution of jobs
> and
> > > > schedules work on TaskManagers.
> > > >
> > > > Quite some parts in the code and documentation/configuration options
> > > > still use that older meaning of JobManager. Newer parts of the
> > > > documentation use "Flink Master" instead.
> > > >
> > > > I'm proposing to go back to calling the orchestrating component
> > > > JobManager, which would mean that we have to touch up the
> documentation
> > > > to remove mentions of "Flink Master". I'm also proposing not to
> mention
> > > > the internal components such as resource manager and dispatcher in
> the
> > > > glossary because there are transparent to users.
> > > >
> > > > I'm proposing to go back to JobManager instead of an alternative name
> > > > also because switching to yet another name would mean many more
> changes
> > > > to code/documentation/peoples minds.
> > > >
> > > > What do you all think?
> > > >
> > > > Best,
> > > > Aljoscha
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-18209
> > > > [2]
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
> > > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable
> > >
> > > https://github.com/knaufk
> > >
> >
>

[Aljoscha Krettek-2]

Thanks for the support! I created a Jira issue and PR:

  - https://issues.apache.org/jira/browse/FLINK-18377
  - https://github.com/apache/flink/pull/12723

Best,
Aljoscha

On 18.06.20 16:28, Marta Paes Moreira wrote:

> +1
>
> I've found the term Flink Master a tad bit confusing myself, in the past,
> as it's not used consistently throughout the documentation (as you mention).
>
> Thanks for following up on this, Aljoscha!
>
> On Wed, Jun 17, 2020 at 5:16 PM Robert Metzger <[hidden email]> wrote:
>
>> Thanks a lot for looking into this!
>>
>> +1 to your proposal
>>
>> On Wed, Jun 17, 2020 at 10:55 AM David Anderson <[hidden email]>
>> wrote:
>>
>>> Aljoscha,
>>>
>>> I think this is a step in the right direction.
>>>
>>> In some cases it may be difficult to talk concretely about the
>>> differences between different deployment models (e.g., comparing a k8s
>>> per-job cluster to a YARN-based session cluster, which is something I
>>> typically present during training) without giving names to the internal
>>> components. I'm not convinced we can completely avoid mentioning the
>>> JobMaster (and Dispatcher and ResourceManagers) in some (rare) contexts
>> --
>>> but I don't see this as an argument against the proposed change.
>>>
>>> David
>>>
>>> On Mon, Jun 15, 2020 at 2:32 PM Konstantin Knauf <[hidden email]>
>>> wrote:
>>>
>>>> Hi Aljoscha,
>>>>
>>>> sounds good to me. Let’s also make sure we don’t refer to the JobMaster
>>> as
>>>> Jobmanager anywhere then (code, config).
>>>>
>>>> I am not sure we can avoid mentioning the Flink ResourceManagers in
>> user
>>>> facing docs completely. For JobMaster and Dispatcher this seems doable.
>>>>
>>>> Best,
>>>>
>>>> Konstantin
>>>>
>>>> On Mon 15. Jun 2020 at 12:56, Aljoscha Krettek <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> This came to my mind because of the master/slave discussion in [1]
>> and
>>>>> the larger discussions about inequality/civil rights happening right
>>> now
>>>>> in the world. I think for this reason alone we should use a name that
>>>>> does not include "master".
>>>>>
>>>>> We could rename it back to JobManager, which was the name mostly used
>>>>> before 2019. Since the beginning of Flink, TaskManager was the term
>>> used
>>>>> for the worker component/node and JobManager was the term used for
>> the
>>>>> orchestrating component/node.
>>>>>
>>>>> Currently our glossary [2] defines these terms (paraphrased by me):
>>>>>
>>>>>    - "Flink Master": it's the orchestrating component that consists of
>>>>> resource manager, dispatcher, and JobManager
>>>>>
>>>>>    - JobManager: it's the thing that manages a single job and runs as
>>>>> part of a "Flink Master"
>>>>>
>>>>>    - TaskManager: it's the worker process
>>>>>
>>>>> Prior to the introduction of the glossary the definition of
>> JobManager
>>>>> would have been:
>>>>>
>>>>>    - It's the orchestrating component that manages execution of jobs
>> and
>>>>> schedules work on TaskManagers.
>>>>>
>>>>> Quite some parts in the code and documentation/configuration options
>>>>> still use that older meaning of JobManager. Newer parts of the
>>>>> documentation use "Flink Master" instead.
>>>>>
>>>>> I'm proposing to go back to calling the orchestrating component
>>>>> JobManager, which would mean that we have to touch up the
>> documentation
>>>>> to remove mentions of "Flink Master". I'm also proposing not to
>> mention
>>>>> the internal components such as resource manager and dispatcher in
>> the
>>>>> glossary because there are transparent to users.
>>>>>
>>>>> I'm proposing to go back to JobManager instead of an alternative name
>>>>> also because switching to yet another name would mean many more
>> changes
>>>>> to code/documentation/peoples minds.
>>>>>
>>>>> What do you all think?
>>>>>
>>>>> Best,
>>>>> Aljoscha
>>>>>
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-18209
>>>>> [2]
>>>>>
>>>>>
>>>>
>>>
>> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
>>>>>
>>>> --
>>>>
>>>> Konstantin Knauf
>>>>
>>>> https://twitter.com/snntrable
>>>>
>>>> https://github.com/knaufk
>>>>
>>>
>>
>