[Discussion] Move cluster creation functions from flink-cli.sh to other scripts

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Discussion] Move cluster creation functions from flink-cli.sh to other scripts

Renjie Liu
Hi:
Flink 1.5.0 brings new deployment mode and we are able to provid a single
cluster for our jobs which manages resources elastically. However, this
also brings challenges for our cluster management since users can submit
jobs or create yarn cluster in the same flink-cli.sh.
Here is the motivation case for this issue:

1. Admins deployed a mesos session cluster using zookeeper ha mode
2. End user can submit jobs to this cluster
3. User can also create yarn cluster using the same conf mistakenly, which
brings our mesos session cluster down

I propose to separate cluster manager functionalities from flink-cli.sh,
and put them into other scripts.
--
Liu, Renjie
Software Engineer, MVAD
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Move cluster creation functions from flink-cli.sh to other scripts

Till Rohrmann
Hi Renjie,

do you mean to not support deploying Flink clusters via
`FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job
cluster for a given job that way is quite useful.

Cheers,
Till

On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <[hidden email]> wrote:

> Hi:
> Flink 1.5.0 brings new deployment mode and we are able to provid a single
> cluster for our jobs which manages resources elastically. However, this
> also brings challenges for our cluster management since users can submit
> jobs or create yarn cluster in the same flink-cli.sh.
> Here is the motivation case for this issue:
>
> 1. Admins deployed a mesos session cluster using zookeeper ha mode
> 2. End user can submit jobs to this cluster
> 3. User can also create yarn cluster using the same conf mistakenly, which
> brings our mesos session cluster down
>
> I propose to separate cluster manager functionalities from flink-cli.sh,
> and put them into other scripts.
> --
> Liu, Renjie
> Software Engineer, MVAD
>
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Move cluster creation functions from flink-cli.sh to other scripts

Renjie Liu
But it's also dangerous if user need to submit job via the same cluster
configuration. I mean we can use another script for starting job cluster,
so that cluster administrators can separate environments.
On Tue, Aug 21, 2018 at 5:53 PM Till Rohrmann <[hidden email]> wrote:

> Hi Renjie,
>
> do you mean to not support deploying Flink clusters via
> `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job
> cluster for a given job that way is quite useful.
>
> Cheers,
> Till
>
> On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <[hidden email]> wrote:
>
> > Hi:
> > Flink 1.5.0 brings new deployment mode and we are able to provid a single
> > cluster for our jobs which manages resources elastically. However, this
> > also brings challenges for our cluster management since users can submit
> > jobs or create yarn cluster in the same flink-cli.sh.
> > Here is the motivation case for this issue:
> >
> > 1. Admins deployed a mesos session cluster using zookeeper ha mode
> > 2. End user can submit jobs to this cluster
> > 3. User can also create yarn cluster using the same conf mistakenly,
> which
> > brings our mesos session cluster down
> >
> > I propose to separate cluster manager functionalities from flink-cli.sh,
> > and put them into other scripts.
> > --
> > Liu, Renjie
> > Software Engineer, MVAD
> >
>
--
Liu, Renjie
Software Engineer, MVAD
Reply | Threaded
Open this post in threaded view
|

Re: [Discussion] Move cluster creation functions from flink-cli.sh to other scripts

Till Rohrmann
How would that look in detail given that I am a user who wants to start a
job cluster?

On Tue, Aug 21, 2018 at 12:09 PM Renjie Liu <[hidden email]> wrote:

> But it's also dangerous if user need to submit job via the same cluster
> configuration. I mean we can use another script for starting job cluster,
> so that cluster administrators can separate environments.
> On Tue, Aug 21, 2018 at 5:53 PM Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi Renjie,
> >
> > do you mean to not support deploying Flink clusters via
> > `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job
> > cluster for a given job that way is quite useful.
> >
> > Cheers,
> > Till
> >
> > On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <[hidden email]>
> wrote:
> >
> > > Hi:
> > > Flink 1.5.0 brings new deployment mode and we are able to provid a
> single
> > > cluster for our jobs which manages resources elastically. However, this
> > > also brings challenges for our cluster management since users can
> submit
> > > jobs or create yarn cluster in the same flink-cli.sh.
> > > Here is the motivation case for this issue:
> > >
> > > 1. Admins deployed a mesos session cluster using zookeeper ha mode
> > > 2. End user can submit jobs to this cluster
> > > 3. User can also create yarn cluster using the same conf mistakenly,
> > which
> > > brings our mesos session cluster down
> > >
> > > I propose to separate cluster manager functionalities from
> flink-cli.sh,
> > > and put them into other scripts.
> > > --
> > > Liu, Renjie
> > > Software Engineer, MVAD
> > >
> >
> --
> Liu, Renjie
> Software Engineer, MVAD
>