Hi:
Flink 1.5.0 brings new deployment mode and we are able to provid a single cluster for our jobs which manages resources elastically. However, this also brings challenges for our cluster management since users can submit jobs or create yarn cluster in the same flink-cli.sh. Here is the motivation case for this issue: 1. Admins deployed a mesos session cluster using zookeeper ha mode 2. End user can submit jobs to this cluster 3. User can also create yarn cluster using the same conf mistakenly, which brings our mesos session cluster down I propose to separate cluster manager functionalities from flink-cli.sh, and put them into other scripts. -- Liu, Renjie Software Engineer, MVAD |
Hi Renjie,
do you mean to not support deploying Flink clusters via `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job cluster for a given job that way is quite useful. Cheers, Till On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <[hidden email]> wrote: > Hi: > Flink 1.5.0 brings new deployment mode and we are able to provid a single > cluster for our jobs which manages resources elastically. However, this > also brings challenges for our cluster management since users can submit > jobs or create yarn cluster in the same flink-cli.sh. > Here is the motivation case for this issue: > > 1. Admins deployed a mesos session cluster using zookeeper ha mode > 2. End user can submit jobs to this cluster > 3. User can also create yarn cluster using the same conf mistakenly, which > brings our mesos session cluster down > > I propose to separate cluster manager functionalities from flink-cli.sh, > and put them into other scripts. > -- > Liu, Renjie > Software Engineer, MVAD > |
But it's also dangerous if user need to submit job via the same cluster
configuration. I mean we can use another script for starting job cluster, so that cluster administrators can separate environments. On Tue, Aug 21, 2018 at 5:53 PM Till Rohrmann <[hidden email]> wrote: > Hi Renjie, > > do you mean to not support deploying Flink clusters via > `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job > cluster for a given job that way is quite useful. > > Cheers, > Till > > On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <[hidden email]> wrote: > > > Hi: > > Flink 1.5.0 brings new deployment mode and we are able to provid a single > > cluster for our jobs which manages resources elastically. However, this > > also brings challenges for our cluster management since users can submit > > jobs or create yarn cluster in the same flink-cli.sh. > > Here is the motivation case for this issue: > > > > 1. Admins deployed a mesos session cluster using zookeeper ha mode > > 2. End user can submit jobs to this cluster > > 3. User can also create yarn cluster using the same conf mistakenly, > which > > brings our mesos session cluster down > > > > I propose to separate cluster manager functionalities from flink-cli.sh, > > and put them into other scripts. > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
How would that look in detail given that I am a user who wants to start a
job cluster? On Tue, Aug 21, 2018 at 12:09 PM Renjie Liu <[hidden email]> wrote: > But it's also dangerous if user need to submit job via the same cluster > configuration. I mean we can use another script for starting job cluster, > so that cluster administrators can separate environments. > On Tue, Aug 21, 2018 at 5:53 PM Till Rohrmann <[hidden email]> > wrote: > > > Hi Renjie, > > > > do you mean to not support deploying Flink clusters via > > `FLINK_HOME/bin/flink -m yarn-cluster`? I think starting a dedicated job > > cluster for a given job that way is quite useful. > > > > Cheers, > > Till > > > > On Thu, Aug 9, 2018 at 4:33 AM Renjie Liu <[hidden email]> > wrote: > > > > > Hi: > > > Flink 1.5.0 brings new deployment mode and we are able to provid a > single > > > cluster for our jobs which manages resources elastically. However, this > > > also brings challenges for our cluster management since users can > submit > > > jobs or create yarn cluster in the same flink-cli.sh. > > > Here is the motivation case for this issue: > > > > > > 1. Admins deployed a mesos session cluster using zookeeper ha mode > > > 2. End user can submit jobs to this cluster > > > 3. User can also create yarn cluster using the same conf mistakenly, > > which > > > brings our mesos session cluster down > > > > > > I propose to separate cluster manager functionalities from > flink-cli.sh, > > > and put them into other scripts. > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
Free forum by Nabble | Edit this page |