[DISCUSS] Expose multiple level clients

[tison]

Hi all,

While drafting FLIP-74 I notice that a job level client(called JobClient)
is always retrieved from a
Flink application cluster level client(called ClusterClient), which is then
always retrieved from a
extern cluster(YARN, mesos, k8s, etc.) level client(called
ClusterDescriptor).

A Flink job management platform possibly requires all levels of client
mentioned above for
customized Flink cluster deployment, Flink cluster management, job
submission and job
management. They can be sorted into interfaces of different level of client
mentioned above.

ClusterDescriptor: Flink cluster deployment
ClusterClient(retrieved from ClusterDescriptor): Flink cluster management
and job submission
JobClient(retrieved from ClusterClient): job management

Recently we have FLIP-73 and FLIP-74 working on client API side, FLIP-74 is
aimed at the job
level client called JobClient which take responsibility of job management.
FLIP-73 is aimed at
a dedicated job executor which take responsibility of job submission.
However, for full functions to
handle Flink clusters & jobs, Flink cluster management and Flink cluster
deployment still require
public interface.These interface is mainly used for downstream project
developer instead of users
who are only interested in Flink job.

Further, we already used ClusterClient and ClusterDescriptor in
CliFrontend, sql-client,
scala-shell and so on, which can be regarded as downstream project hosted
in Flink repo.
Given this observation, there is no reason we don't expose API for Flink
cluster management
and Flink cluster deployment.

So here comes two question:

1. Do we want to expose multiple level clients as described above? If no,
why? If so, what does
the plan look like?
2. Does Executor introduced in the ongoing FLIP-73 break multiple level
clients layout
described above? Since ClusterClient already implements functions for job
submission, where
is Executor in the layout above? Does it overlap with existing client
concept?

Best,
tison.

[Kostas Kloudas-4]

Hi Tison,

I shared some thoughts on this topic in the FLIP-74 thread, as some of
the questions are related to that.

Cheers,
Kostas

On Wed, Sep 25, 2019 at 12:30 PM Zili Chen <[hidden email]> wrote:

>
> Hi all,
>
> While drafting FLIP-74 I notice that a job level client(called JobClient) is always retrieved from a
> Flink application cluster level client(called ClusterClient), which is then always retrieved from a
> extern cluster(YARN, mesos, k8s, etc.) level client(called ClusterDescriptor).
>
> A Flink job management platform possibly requires all levels of client mentioned above for
> customized Flink cluster deployment, Flink cluster management, job submission and job
> management. They can be sorted into interfaces of different level of client mentioned above.
>
> ClusterDescriptor: Flink cluster deployment
> ClusterClient(retrieved from ClusterDescriptor): Flink cluster management and job submission
> JobClient(retrieved from ClusterClient): job management
>
> Recently we have FLIP-73 and FLIP-74 working on client API side, FLIP-74 is aimed at the job
> level client called JobClient which take responsibility of job management. FLIP-73 is aimed at
> a dedicated job executor which take responsibility of job submission. However, for full functions to
> handle Flink clusters & jobs, Flink cluster management and Flink cluster deployment still require
> public interface.These interface is mainly used for downstream project developer instead of users
> who are only interested in Flink job.
>
> Further, we already used ClusterClient and ClusterDescriptor in CliFrontend, sql-client,
> scala-shell and so on, which can be regarded as downstream project hosted in Flink repo.
> Given this observation, there is no reason we don't expose API for Flink cluster management
> and Flink cluster deployment.
>
> So here comes two question:
>
> 1. Do we want to expose multiple level clients as described above? If no, why? If so, what does
> the plan look like?
> 2. Does Executor introduced in the ongoing FLIP-73 break multiple level clients layout
> described above? Since ClusterClient already implements functions for job submission, where
> is Executor in the layout above? Does it overlap with existing client concept?
>
> Best,
> tison.

[tison]

Thanks for your reply Kostas!

Sorry for replying late. The most unclear point about multi-layered client
is about where Executor is.
I will also separate the corresponding reply into FLIP-73 and FLIP-74
dedicated thread.

Best,
tison.


Kostas Kloudas <[hidden email]> 于2019年9月25日周三 下午9:31写道：

> Hi Tison,
>
> I shared some thoughts on this topic in the FLIP-74 thread, as some of
> the questions are related to that.
>
> Cheers,
> Kostas
>
> On Wed, Sep 25, 2019 at 12:30 PM Zili Chen <[hidden email]> wrote:
> >
> > Hi all,
> >
> > While drafting FLIP-74 I notice that a job level client(called
> JobClient) is always retrieved from a
> > Flink application cluster level client(called ClusterClient), which is
> then always retrieved from a
> > extern cluster(YARN, mesos, k8s, etc.) level client(called
> ClusterDescriptor).
> >
> > A Flink job management platform possibly requires all levels of client
> mentioned above for
> > customized Flink cluster deployment, Flink cluster management, job
> submission and job
> > management. They can be sorted into interfaces of different level of
> client mentioned above.
> >
> > ClusterDescriptor: Flink cluster deployment
> > ClusterClient(retrieved from ClusterDescriptor): Flink cluster
> management and job submission
> > JobClient(retrieved from ClusterClient): job management
> >
> > Recently we have FLIP-73 and FLIP-74 working on client API side, FLIP-74
> is aimed at the job
> > level client called JobClient which take responsibility of job
> management. FLIP-73 is aimed at
> > a dedicated job executor which take responsibility of job submission.
> However, for full functions to
> > handle Flink clusters & jobs, Flink cluster management and Flink cluster
> deployment still require
> > public interface.These interface is mainly used for downstream project
> developer instead of users
> > who are only interested in Flink job.
> >
> > Further, we already used ClusterClient and ClusterDescriptor in
> CliFrontend, sql-client,
> > scala-shell and so on, which can be regarded as downstream project
> hosted in Flink repo.
> > Given this observation, there is no reason we don't expose API for Flink
> cluster management
> > and Flink cluster deployment.
> >
> > So here comes two question:
> >
> > 1. Do we want to expose multiple level clients as described above? If
> no, why? If so, what does
> > the plan look like?
> > 2. Does Executor introduced in the ongoing FLIP-73 break multiple level
> clients layout
> > described above? Since ClusterClient already implements functions for
> job submission, where
> > is Executor in the layout above? Does it overlap with existing client
> concept?
> >
> > Best,
> > tison.
>