Fwd: [DISCUSS] FLIP-6 Problems
Locked
 
|
---------- Forwarded message ---------
From: Renjie Liu <[hidden email]> Date: Tue, Jun 5, 2018 at 10:43 AM Subject: [DISCUSS] FLIP-6 Problems To: user <[hidden email]> Hi: We've deployed flink 1.5.0 and tested the new cluster manager, it's really great for flink to be elastic. However we've also found some problems that blocks us from deploying it to production environment. 1. Task manager isolation. Currently flink allows different jobs to execute on same task managers, this is unacceptable in production environment since a faulty written job may kill task managers and affect other jobs. 2. Per job resource configuration. Currently flink session cluster can only allocate same size and configuration task managers. This may waste a lot of resources if we have a lot of jobs with different resource requirement. 3. Task manager's name is meanless. This is a problem since we can't monitor status of container in mesos environment. One solution to the above problems is to use per job cluster, but a centralized cluster manager can help to manage flink deployment and jobs better. How you guys think about those? If the community agrees with us, we would like to propose design and implementation to enhance the flink cluster manager. -- Liu, Renjie Software Engineer, MVAD -- Liu, Renjie Software Engineer, MVAD |
|
Hi Renjie,
1) you're right that the Flink session mode does not give you proper job isolation. It is the same as with Flink 1.4 session mode. If this is a strong requirement for you, then I recommend using the per job mode. 2) At the moment it is also not possible to define per job resource requirements when using the session mode. This is a feature which the community has started implementing but it is not yet fully done. I assume that the community will continue working on it. At the moment, the solution would be to use the per job mode to not waste unnecessary resources. 3) I think the assigned ResourceID for a TaskManager is shown in the web UI and when querying the "/taskmanagers" REST endpoint. The resource id is derived from the Mesos task id. Would that help to identify which TM is running on which Mesos task? Cheers, Till On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu <[hidden email]> wrote: > ---------- Forwarded message --------- > From: Renjie Liu <[hidden email]> > Date: Tue, Jun 5, 2018 at 10:43 AM > Subject: [DISCUSS] FLIP-6 Problems > To: user <[hidden email]> > > > Hi: > > We've deployed flink 1.5.0 and tested the new cluster manager, it's really > great for flink to be elastic. However we've also found some problems that > blocks us from deploying it to production environment. > > 1. Task manager isolation. Currently flink allows different jobs to execute > on same task managers, this is unacceptable in production environment since > a faulty written job may kill task managers and affect other jobs. > 2. Per job resource configuration. Currently flink session cluster can only > allocate same size and configuration task managers. This may waste a lot of > resources if we have a lot of jobs with different resource requirement. > 3. Task manager's name is meanless. This is a problem since we can't > monitor status of container in mesos environment. > > One solution to the above problems is to use per job cluster, but a > centralized cluster manager can help to manage flink deployment and jobs > better. > > How you guys think about those? If the community agrees with us, we would > like to propose design and implementation to enhance the flink cluster > manager. > -- > Liu, Renjie > Software Engineer, MVAD > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Hi, Till:
1. Does the community has any plan to add task manager isolation into the session mode? 2. Is there any issues to track this feature? I want to help contribute. 3. Thanks for the knowledge but it can't help if task manager isolation is not present. On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann <[hidden email]> wrote: > Hi Renjie, > > 1) you're right that the Flink session mode does not give you proper job > isolation. It is the same as with Flink 1.4 session mode. If this is a > strong requirement for you, then I recommend using the per job mode. > > 2) At the moment it is also not possible to define per job resource > requirements when using the session mode. This is a feature which the > community has started implementing but it is not yet fully done. I assume > that the community will continue working on it. At the moment, the solution > would be to use the per job mode to not waste unnecessary resources. > > 3) I think the assigned ResourceID for a TaskManager is shown in the web UI > and when querying the "/taskmanagers" REST endpoint. The resource id is > derived from the Mesos task id. Would that help to identify which TM is > running on which Mesos task? > > Cheers, > Till > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu <[hidden email]> wrote: > > > ---------- Forwarded message --------- > > From: Renjie Liu <[hidden email]> > > Date: Tue, Jun 5, 2018 at 10:43 AM > > Subject: [DISCUSS] FLIP-6 Problems > > To: user <[hidden email]> > > > > > > Hi: > > > > We've deployed flink 1.5.0 and tested the new cluster manager, it's > really > > great for flink to be elastic. However we've also found some problems > that > > blocks us from deploying it to production environment. > > > > 1. Task manager isolation. Currently flink allows different jobs to > execute > > on same task managers, this is unacceptable in production environment > since > > a faulty written job may kill task managers and affect other jobs. > > 2. Per job resource configuration. Currently flink session cluster can > only > > allocate same size and configuration task managers. This may waste a lot > of > > resources if we have a lot of jobs with different resource requirement. > > 3. Task manager's name is meanless. This is a problem since we can't > > monitor status of container in mesos environment. > > > > One solution to the above problems is to use per job cluster, but a > > centralized cluster manager can help to manage flink deployment and jobs > > better. > > > > How you guys think about those? If the community agrees with us, we would > > like to propose design and implementation to enhance the flink cluster > > manager. > > -- > > Liu, Renjie > > Software Engineer, MVAD > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
Hi Renjie,
When you suggest to have TaskManager isolation in session mode, do you mean to have a shared JobManager / Dispatcher, but job-specific TaskManagers? If this mainly to reduce the overhead of the per-job JobManager? One assumption so far was that if TaskManager isolation is required, JobManager isolation is also required, because some user code potentially also runs on the JobManager, like CheckpointHooks, Input/Output Formats, ... Best, Stephan On Tue, Jun 5, 2018 at 4:20 PM, Renjie Liu <[hidden email]> wrote: > Hi, Till: > > > 1. Does the community has any plan to add task manager isolation into > the session mode? > 2. Is there any issues to track this feature? I want to help contribute. > 3. Thanks for the knowledge but it can't help if task manager isolation > is not present. > > > On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann <[hidden email]> wrote: > > > Hi Renjie, > > > > 1) you're right that the Flink session mode does not give you proper job > > isolation. It is the same as with Flink 1.4 session mode. If this is a > > strong requirement for you, then I recommend using the per job mode. > > > > 2) At the moment it is also not possible to define per job resource > > requirements when using the session mode. This is a feature which the > > community has started implementing but it is not yet fully done. I assume > > that the community will continue working on it. At the moment, the > solution > > would be to use the per job mode to not waste unnecessary resources. > > > > 3) I think the assigned ResourceID for a TaskManager is shown in the web > UI > > and when querying the "/taskmanagers" REST endpoint. The resource id is > > derived from the Mesos task id. Would that help to identify which TM is > > running on which Mesos task? > > > > Cheers, > > Till > > > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu <[hidden email]> > wrote: > > > > > ---------- Forwarded message --------- > > > From: Renjie Liu <[hidden email]> > > > Date: Tue, Jun 5, 2018 at 10:43 AM > > > Subject: [DISCUSS] FLIP-6 Problems > > > To: user <[hidden email]> > > > > > > > > > Hi: > > > > > > We've deployed flink 1.5.0 and tested the new cluster manager, it's > > really > > > great for flink to be elastic. However we've also found some problems > > that > > > blocks us from deploying it to production environment. > > > > > > 1. Task manager isolation. Currently flink allows different jobs to > > execute > > > on same task managers, this is unacceptable in production environment > > since > > > a faulty written job may kill task managers and affect other jobs. > > > 2. Per job resource configuration. Currently flink session cluster can > > only > > > allocate same size and configuration task managers. This may waste a > lot > > of > > > resources if we have a lot of jobs with different resource requirement. > > > 3. Task manager's name is meanless. This is a problem since we can't > > > monitor status of container in mesos environment. > > > > > > One solution to the above problems is to use per job cluster, but a > > > centralized cluster manager can help to manage flink deployment and > jobs > > > better. > > > > > > How you guys think about those? If the community agrees with us, we > would > > > like to propose design and implementation to enhance the flink cluster > > > manager. > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Hi, Stephan:
Yes that's what I mean. In fact the most import thing is to share the dispatcher so that we can have *a centralized gateway for flink job management and submission. The problem with per job cluster is that we can't have a centralized gateway.* I didn't realize that job manager also needs to run user code before and yes that means we job manager should also be isolated. Wouldn't it be better to separate job manager from the dispatcher so that user code does't interfere with each other? In fact it seems that in most production environments job isolation is required since nobody want their job to be affected by others. On Tue, Jun 5, 2018 at 11:34 PM Stephan Ewen <[hidden email]> wrote: > Hi Renjie, > > When you suggest to have TaskManager isolation in session mode, do you mean > to have a shared JobManager / Dispatcher, but job-specific TaskManagers? > If this mainly to reduce the overhead of the per-job JobManager? > > One assumption so far was that if TaskManager isolation is required, > JobManager isolation is also required, because some user code potentially > also runs on the JobManager, like CheckpointHooks, Input/Output Formats, > ... > > Best, > Stephan > > > > On Tue, Jun 5, 2018 at 4:20 PM, Renjie Liu <[hidden email]> > wrote: > > > Hi, Till: > > > > > > 1. Does the community has any plan to add task manager isolation into > > the session mode? > > 2. Is there any issues to track this feature? I want to help > contribute. > > 3. Thanks for the knowledge but it can't help if task manager > isolation > > is not present. > > > > > > On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann <[hidden email]> > wrote: > > > > > Hi Renjie, > > > > > > 1) you're right that the Flink session mode does not give you proper > job > > > isolation. It is the same as with Flink 1.4 session mode. If this is a > > > strong requirement for you, then I recommend using the per job mode. > > > > > > 2) At the moment it is also not possible to define per job resource > > > requirements when using the session mode. This is a feature which the > > > community has started implementing but it is not yet fully done. I > assume > > > that the community will continue working on it. At the moment, the > > solution > > > would be to use the per job mode to not waste unnecessary resources. > > > > > > 3) I think the assigned ResourceID for a TaskManager is shown in the > web > > UI > > > and when querying the "/taskmanagers" REST endpoint. The resource id is > > > derived from the Mesos task id. Would that help to identify which TM is > > > running on which Mesos task? > > > > > > Cheers, > > > Till > > > > > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu <[hidden email]> > > wrote: > > > > > > > ---------- Forwarded message --------- > > > > From: Renjie Liu <[hidden email]> > > > > Date: Tue, Jun 5, 2018 at 10:43 AM > > > > Subject: [DISCUSS] FLIP-6 Problems > > > > To: user <[hidden email]> > > > > > > > > > > > > Hi: > > > > > > > > We've deployed flink 1.5.0 and tested the new cluster manager, it's > > > really > > > > great for flink to be elastic. However we've also found some problems > > > that > > > > blocks us from deploying it to production environment. > > > > > > > > 1. Task manager isolation. Currently flink allows different jobs to > > > execute > > > > on same task managers, this is unacceptable in production environment > > > since > > > > a faulty written job may kill task managers and affect other jobs. > > > > 2. Per job resource configuration. Currently flink session cluster > can > > > only > > > > allocate same size and configuration task managers. This may waste a > > lot > > > of > > > > resources if we have a lot of jobs with different resource > requirement. > > > > 3. Task manager's name is meanless. This is a problem since we can't > > > > monitor status of container in mesos environment. > > > > > > > > One solution to the above problems is to use per job cluster, but a > > > > centralized cluster manager can help to manage flink deployment and > > jobs > > > > better. > > > > > > > > How you guys think about those? If the community agrees with us, we > > would > > > > like to propose design and implementation to enhance the flink > cluster > > > > manager. > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
|
Hi Renjie,
there is already an issue for introducing further scheduling constraints (e.g. tags) to achieve TM isolation when using the session mode [1]. What it does not cover is the isolation of the JMs which need to be executed in their own processes. At the moment they share the same process with the Dispatcher because it was simpler to do it like that as first iteration. Here is the issue for isolating JobManagers [2]. Concerning the resource specification, the corresponding issue can be found here [3]. [1] https://issues.apache.org/jira/browse/FLINK-8886 [2] https://issues.apache.org/jira/browse/FLINK-9537 [3] https://issues.apache.org/jira/browse/FLINK-5131 Cheers, Till On Wed, Jun 6, 2018 at 2:13 AM Renjie Liu <[hidden email]> wrote: > Hi, Stephan: > > Yes that's what I mean. In fact the most import thing is to share the > dispatcher so that we can have *a centralized gateway for flink job > management and submission. The problem with per job cluster is that we > can't have a centralized gateway.* > > I didn't realize that job manager also needs to run user code before and > yes that means we job manager should also be isolated. > > Wouldn't it be better to separate job manager from the dispatcher so that > user code does't interfere with each other? In fact it seems that in most > production environments job isolation is required since nobody want their > job to be affected by others. > > On Tue, Jun 5, 2018 at 11:34 PM Stephan Ewen <[hidden email]> wrote: > > > Hi Renjie, > > > > When you suggest to have TaskManager isolation in session mode, do you > mean > > to have a shared JobManager / Dispatcher, but job-specific TaskManagers? > > If this mainly to reduce the overhead of the per-job JobManager? > > > > One assumption so far was that if TaskManager isolation is required, > > JobManager isolation is also required, because some user code potentially > > also runs on the JobManager, like CheckpointHooks, Input/Output Formats, > > ... > > > > Best, > > Stephan > > > > > > > > On Tue, Jun 5, 2018 at 4:20 PM, Renjie Liu <[hidden email]> > > wrote: > > > > > Hi, Till: > > > > > > > > > 1. Does the community has any plan to add task manager isolation > into > > > the session mode? > > > 2. Is there any issues to track this feature? I want to help > > contribute. > > > 3. Thanks for the knowledge but it can't help if task manager > > isolation > > > is not present. > > > > > > > > > On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann <[hidden email]> > > wrote: > > > > > > > Hi Renjie, > > > > > > > > 1) you're right that the Flink session mode does not give you proper > > job > > > > isolation. It is the same as with Flink 1.4 session mode. If this is > a > > > > strong requirement for you, then I recommend using the per job mode. > > > > > > > > 2) At the moment it is also not possible to define per job resource > > > > requirements when using the session mode. This is a feature which the > > > > community has started implementing but it is not yet fully done. I > > assume > > > > that the community will continue working on it. At the moment, the > > > solution > > > > would be to use the per job mode to not waste unnecessary resources. > > > > > > > > 3) I think the assigned ResourceID for a TaskManager is shown in the > > web > > > UI > > > > and when querying the "/taskmanagers" REST endpoint. The resource id > is > > > > derived from the Mesos task id. Would that help to identify which TM > is > > > > running on which Mesos task? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu <[hidden email]> > > > wrote: > > > > > > > > > ---------- Forwarded message --------- > > > > > From: Renjie Liu <[hidden email]> > > > > > Date: Tue, Jun 5, 2018 at 10:43 AM > > > > > Subject: [DISCUSS] FLIP-6 Problems > > > > > To: user <[hidden email]> > > > > > > > > > > > > > > > Hi: > > > > > > > > > > We've deployed flink 1.5.0 and tested the new cluster manager, it's > > > > really > > > > > great for flink to be elastic. However we've also found some > problems > > > > that > > > > > blocks us from deploying it to production environment. > > > > > > > > > > 1. Task manager isolation. Currently flink allows different jobs to > > > > execute > > > > > on same task managers, this is unacceptable in production > environment > > > > since > > > > > a faulty written job may kill task managers and affect other jobs. > > > > > 2. Per job resource configuration. Currently flink session cluster > > can > > > > only > > > > > allocate same size and configuration task managers. This may waste > a > > > lot > > > > of > > > > > resources if we have a lot of jobs with different resource > > requirement. > > > > > 3. Task manager's name is meanless. This is a problem since we > can't > > > > > monitor status of container in mesos environment. > > > > > > > > > > One solution to the above problems is to use per job cluster, but a > > > > > centralized cluster manager can help to manage flink deployment and > > > jobs > > > > > better. > > > > > > > > > > How you guys think about those? If the community agrees with us, we > > > would > > > > > like to propose design and implementation to enhance the flink > > cluster > > > > > manager. > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
That's really great! I'll help to contribute to the process.
On Wed, Jun 6, 2018 at 3:17 PM Till Rohrmann <[hidden email]> wrote: > Hi Renjie, > > there is already an issue for introducing further scheduling constraints > (e.g. tags) to achieve TM isolation when using the session mode [1]. What > it does not cover is the isolation of the JMs which need to be executed in > their own processes. At the moment they share the same process with the > Dispatcher because it was simpler to do it like that as first iteration. > Here is the issue for isolating JobManagers [2]. > > Concerning the resource specification, the corresponding issue can be found > here [3]. > > [1] https://issues.apache.org/jira/browse/FLINK-8886 > [2] https://issues.apache.org/jira/browse/FLINK-9537 > [3] https://issues.apache.org/jira/browse/FLINK-5131 > > Cheers, > Till > > On Wed, Jun 6, 2018 at 2:13 AM Renjie Liu <[hidden email]> wrote: > > > Hi, Stephan: > > > > Yes that's what I mean. In fact the most import thing is to share the > > dispatcher so that we can have *a centralized gateway for flink job > > management and submission. The problem with per job cluster is that we > > can't have a centralized gateway.* > > > > I didn't realize that job manager also needs to run user code before and > > yes that means we job manager should also be isolated. > > > > Wouldn't it be better to separate job manager from the dispatcher so that > > user code does't interfere with each other? In fact it seems that in most > > production environments job isolation is required since nobody want their > > job to be affected by others. > > > > On Tue, Jun 5, 2018 at 11:34 PM Stephan Ewen <[hidden email]> wrote: > > > > > Hi Renjie, > > > > > > When you suggest to have TaskManager isolation in session mode, do you > > mean > > > to have a shared JobManager / Dispatcher, but job-specific > TaskManagers? > > > If this mainly to reduce the overhead of the per-job JobManager? > > > > > > One assumption so far was that if TaskManager isolation is required, > > > JobManager isolation is also required, because some user code > potentially > > > also runs on the JobManager, like CheckpointHooks, Input/Output > Formats, > > > ... > > > > > > Best, > > > Stephan > > > > > > > > > > > > On Tue, Jun 5, 2018 at 4:20 PM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > > Hi, Till: > > > > > > > > > > > > 1. Does the community has any plan to add task manager isolation > > into > > > > the session mode? > > > > 2. Is there any issues to track this feature? I want to help > > > contribute. > > > > 3. Thanks for the knowledge but it can't help if task manager > > > isolation > > > > is not present. > > > > > > > > > > > > On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > 1) you're right that the Flink session mode does not give you > proper > > > job > > > > > isolation. It is the same as with Flink 1.4 session mode. If this > is > > a > > > > > strong requirement for you, then I recommend using the per job > mode. > > > > > > > > > > 2) At the moment it is also not possible to define per job resource > > > > > requirements when using the session mode. This is a feature which > the > > > > > community has started implementing but it is not yet fully done. I > > > assume > > > > > that the community will continue working on it. At the moment, the > > > > solution > > > > > would be to use the per job mode to not waste unnecessary > resources. > > > > > > > > > > 3) I think the assigned ResourceID for a TaskManager is shown in > the > > > web > > > > UI > > > > > and when querying the "/taskmanagers" REST endpoint. The resource > id > > is > > > > > derived from the Mesos task id. Would that help to identify which > TM > > is > > > > > running on which Mesos task? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu <[hidden email] > > > > > > wrote: > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > From: Renjie Liu <[hidden email]> > > > > > > Date: Tue, Jun 5, 2018 at 10:43 AM > > > > > > Subject: [DISCUSS] FLIP-6 Problems > > > > > > To: user <[hidden email]> > > > > > > > > > > > > > > > > > > Hi: > > > > > > > > > > > > We've deployed flink 1.5.0 and tested the new cluster manager, > it's > > > > > really > > > > > > great for flink to be elastic. However we've also found some > > problems > > > > > that > > > > > > blocks us from deploying it to production environment. > > > > > > > > > > > > 1. Task manager isolation. Currently flink allows different jobs > to > > > > > execute > > > > > > on same task managers, this is unacceptable in production > > environment > > > > > since > > > > > > a faulty written job may kill task managers and affect other > jobs. > > > > > > 2. Per job resource configuration. Currently flink session > cluster > > > can > > > > > only > > > > > > allocate same size and configuration task managers. This may > waste > > a > > > > lot > > > > > of > > > > > > resources if we have a lot of jobs with different resource > > > requirement. > > > > > > 3. Task manager's name is meanless. This is a problem since we > > can't > > > > > > monitor status of container in mesos environment. > > > > > > > > > > > > One solution to the above problems is to use per job cluster, > but a > > > > > > centralized cluster manager can help to manage flink deployment > and > > > > jobs > > > > > > better. > > > > > > > > > > > > How you guys think about those? If the community agrees with us, > we > > > > would > > > > > > like to propose design and implementation to enhance the flink > > > cluster > > > > > > manager. > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
|
The FLIP-6 design was specifically such that it allows for separation of
Dispatcher, ResourceManager, and JobManagers. So that could be another extension at some point. It should be conceptually rather simple, the dispatcher creates per job a new container launch context with the "JobManagerRunner" and starts that. In practice, it is quite a bit of work still, with all the details of Yarn to take care of. On Wed, Jun 6, 2018 at 9:45 AM, Renjie Liu <[hidden email]> wrote: > That's really great! I'll help to contribute to the process. > > On Wed, Jun 6, 2018 at 3:17 PM Till Rohrmann <[hidden email]> wrote: > > > Hi Renjie, > > > > there is already an issue for introducing further scheduling constraints > > (e.g. tags) to achieve TM isolation when using the session mode [1]. What > > it does not cover is the isolation of the JMs which need to be executed > in > > their own processes. At the moment they share the same process with the > > Dispatcher because it was simpler to do it like that as first iteration. > > Here is the issue for isolating JobManagers [2]. > > > > Concerning the resource specification, the corresponding issue can be > found > > here [3]. > > > > [1] https://issues.apache.org/jira/browse/FLINK-8886 > > [2] https://issues.apache.org/jira/browse/FLINK-9537 > > [3] https://issues.apache.org/jira/browse/FLINK-5131 > > > > Cheers, > > Till > > > > On Wed, Jun 6, 2018 at 2:13 AM Renjie Liu <[hidden email]> > wrote: > > > > > Hi, Stephan: > > > > > > Yes that's what I mean. In fact the most import thing is to share the > > > dispatcher so that we can have *a centralized gateway for flink job > > > management and submission. The problem with per job cluster is that we > > > can't have a centralized gateway.* > > > > > > I didn't realize that job manager also needs to run user code before > and > > > yes that means we job manager should also be isolated. > > > > > > Wouldn't it be better to separate job manager from the dispatcher so > that > > > user code does't interfere with each other? In fact it seems that in > most > > > production environments job isolation is required since nobody want > their > > > job to be affected by others. > > > > > > On Tue, Jun 5, 2018 at 11:34 PM Stephan Ewen <[hidden email]> wrote: > > > > > > > Hi Renjie, > > > > > > > > When you suggest to have TaskManager isolation in session mode, do > you > > > mean > > > > to have a shared JobManager / Dispatcher, but job-specific > > TaskManagers? > > > > If this mainly to reduce the overhead of the per-job JobManager? > > > > > > > > One assumption so far was that if TaskManager isolation is required, > > > > JobManager isolation is also required, because some user code > > potentially > > > > also runs on the JobManager, like CheckpointHooks, Input/Output > > Formats, > > > > ... > > > > > > > > Best, > > > > Stephan > > > > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 4:20 PM, Renjie Liu <[hidden email]> > > > > wrote: > > > > > > > > > Hi, Till: > > > > > > > > > > > > > > > 1. Does the community has any plan to add task manager isolation > > > into > > > > > the session mode? > > > > > 2. Is there any issues to track this feature? I want to help > > > > contribute. > > > > > 3. Thanks for the knowledge but it can't help if task manager > > > > isolation > > > > > is not present. > > > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann <[hidden email] > > > > > > wrote: > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > 1) you're right that the Flink session mode does not give you > > proper > > > > job > > > > > > isolation. It is the same as with Flink 1.4 session mode. If this > > is > > > a > > > > > > strong requirement for you, then I recommend using the per job > > mode. > > > > > > > > > > > > 2) At the moment it is also not possible to define per job > resource > > > > > > requirements when using the session mode. This is a feature which > > the > > > > > > community has started implementing but it is not yet fully done. > I > > > > assume > > > > > > that the community will continue working on it. At the moment, > the > > > > > solution > > > > > > would be to use the per job mode to not waste unnecessary > > resources. > > > > > > > > > > > > 3) I think the assigned ResourceID for a TaskManager is shown in > > the > > > > web > > > > > UI > > > > > > and when querying the "/taskmanagers" REST endpoint. The resource > > id > > > is > > > > > > derived from the Mesos task id. Would that help to identify which > > TM > > > is > > > > > > running on which Mesos task? > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > > From: Renjie Liu <[hidden email]> > > > > > > > Date: Tue, Jun 5, 2018 at 10:43 AM > > > > > > > Subject: [DISCUSS] FLIP-6 Problems > > > > > > > To: user <[hidden email]> > > > > > > > > > > > > > > > > > > > > > Hi: > > > > > > > > > > > > > > We've deployed flink 1.5.0 and tested the new cluster manager, > > it's > > > > > > really > > > > > > > great for flink to be elastic. However we've also found some > > > problems > > > > > > that > > > > > > > blocks us from deploying it to production environment. > > > > > > > > > > > > > > 1. Task manager isolation. Currently flink allows different > jobs > > to > > > > > > execute > > > > > > > on same task managers, this is unacceptable in production > > > environment > > > > > > since > > > > > > > a faulty written job may kill task managers and affect other > > jobs. > > > > > > > 2. Per job resource configuration. Currently flink session > > cluster > > > > can > > > > > > only > > > > > > > allocate same size and configuration task managers. This may > > waste > > > a > > > > > lot > > > > > > of > > > > > > > resources if we have a lot of jobs with different resource > > > > requirement. > > > > > > > 3. Task manager's name is meanless. This is a problem since we > > > can't > > > > > > > monitor status of container in mesos environment. > > > > > > > > > > > > > > One solution to the above problems is to use per job cluster, > > but a > > > > > > > centralized cluster manager can help to manage flink deployment > > and > > > > > jobs > > > > > > > better. > > > > > > > > > > > > > > How you guys think about those? If the community agrees with > us, > > we > > > > > would > > > > > > > like to propose design and implementation to enhance the flink > > > > cluster > > > > > > > manager. > > > > > > > -- > > > > > > > Liu, Renjie > > > > > > > Software Engineer, MVAD > > > > > > > -- > > > > > > > Liu, Renjie > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Our main use cases are mesos, maybe we can start with mesos support.
On Wed, Jun 6, 2018 at 5:00 PM Stephan Ewen <[hidden email]> wrote: > The FLIP-6 design was specifically such that it allows for separation of > Dispatcher, ResourceManager, and JobManagers. > So that could be another extension at some point. > > It should be conceptually rather simple, the dispatcher creates per job a > new container launch context with the "JobManagerRunner" and starts that. > In practice, it is quite a bit of work still, with all the details of Yarn > to take care of. > > > > On Wed, Jun 6, 2018 at 9:45 AM, Renjie Liu <[hidden email]> > wrote: > > > That's really great! I'll help to contribute to the process. > > > > On Wed, Jun 6, 2018 at 3:17 PM Till Rohrmann <[hidden email]> > wrote: > > > > > Hi Renjie, > > > > > > there is already an issue for introducing further scheduling > constraints > > > (e.g. tags) to achieve TM isolation when using the session mode [1]. > What > > > it does not cover is the isolation of the JMs which need to be executed > > in > > > their own processes. At the moment they share the same process with the > > > Dispatcher because it was simpler to do it like that as first > iteration. > > > Here is the issue for isolating JobManagers [2]. > > > > > > Concerning the resource specification, the corresponding issue can be > > found > > > here [3]. > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-8886 > > > [2] https://issues.apache.org/jira/browse/FLINK-9537 > > > [3] https://issues.apache.org/jira/browse/FLINK-5131 > > > > > > Cheers, > > > Till > > > > > > On Wed, Jun 6, 2018 at 2:13 AM Renjie Liu <[hidden email]> > > wrote: > > > > > > > Hi, Stephan: > > > > > > > > Yes that's what I mean. In fact the most import thing is to share the > > > > dispatcher so that we can have *a centralized gateway for flink job > > > > management and submission. The problem with per job cluster is that > we > > > > can't have a centralized gateway.* > > > > > > > > I didn't realize that job manager also needs to run user code before > > and > > > > yes that means we job manager should also be isolated. > > > > > > > > Wouldn't it be better to separate job manager from the dispatcher so > > that > > > > user code does't interfere with each other? In fact it seems that in > > most > > > > production environments job isolation is required since nobody want > > their > > > > job to be affected by others. > > > > > > > > On Tue, Jun 5, 2018 at 11:34 PM Stephan Ewen <[hidden email]> > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > When you suggest to have TaskManager isolation in session mode, do > > you > > > > mean > > > > > to have a shared JobManager / Dispatcher, but job-specific > > > TaskManagers? > > > > > If this mainly to reduce the overhead of the per-job JobManager? > > > > > > > > > > One assumption so far was that if TaskManager isolation is > required, > > > > > JobManager isolation is also required, because some user code > > > potentially > > > > > also runs on the JobManager, like CheckpointHooks, Input/Output > > > Formats, > > > > > ... > > > > > > > > > > Best, > > > > > Stephan > > > > > > > > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 4:20 PM, Renjie Liu < > [hidden email]> > > > > > wrote: > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > > > > > > > 1. Does the community has any plan to add task manager > isolation > > > > into > > > > > > the session mode? > > > > > > 2. Is there any issues to track this feature? I want to help > > > > > contribute. > > > > > > 3. Thanks for the knowledge but it can't help if task manager > > > > > isolation > > > > > > is not present. > > > > > > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 7:28 PM Till Rohrmann < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > 1) you're right that the Flink session mode does not give you > > > proper > > > > > job > > > > > > > isolation. It is the same as with Flink 1.4 session mode. If > this > > > is > > > > a > > > > > > > strong requirement for you, then I recommend using the per job > > > mode. > > > > > > > > > > > > > > 2) At the moment it is also not possible to define per job > > resource > > > > > > > requirements when using the session mode. This is a feature > which > > > the > > > > > > > community has started implementing but it is not yet fully > done. > > I > > > > > assume > > > > > > > that the community will continue working on it. At the moment, > > the > > > > > > solution > > > > > > > would be to use the per job mode to not waste unnecessary > > > resources. > > > > > > > > > > > > > > 3) I think the assigned ResourceID for a TaskManager is shown > in > > > the > > > > > web > > > > > > UI > > > > > > > and when querying the "/taskmanagers" REST endpoint. The > resource > > > id > > > > is > > > > > > > derived from the Mesos task id. Would that help to identify > which > > > TM > > > > is > > > > > > > running on which Mesos task? > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 5:13 AM Renjie Liu < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > > > > > From: Renjie Liu <[hidden email]> > > > > > > > > Date: Tue, Jun 5, 2018 at 10:43 AM > > > > > > > > Subject: [DISCUSS] FLIP-6 Problems > > > > > > > > To: user <[hidden email]> > > > > > > > > > > > > > > > > > > > > > > > > Hi: > > > > > > > > > > > > > > > > We've deployed flink 1.5.0 and tested the new cluster > manager, > > > it's > > > > > > > really > > > > > > > > great for flink to be elastic. However we've also found some > > > > problems > > > > > > > that > > > > > > > > blocks us from deploying it to production environment. > > > > > > > > > > > > > > > > 1. Task manager isolation. Currently flink allows different > > jobs > > > to > > > > > > > execute > > > > > > > > on same task managers, this is unacceptable in production > > > > environment > > > > > > > since > > > > > > > > a faulty written job may kill task managers and affect other > > > jobs. > > > > > > > > 2. Per job resource configuration. Currently flink session > > > cluster > > > > > can > > > > > > > only > > > > > > > > allocate same size and configuration task managers. This may > > > waste > > > > a > > > > > > lot > > > > > > > of > > > > > > > > resources if we have a lot of jobs with different resource > > > > > requirement. > > > > > > > > 3. Task manager's name is meanless. This is a problem since > we > > > > can't > > > > > > > > monitor status of container in mesos environment. > > > > > > > > > > > > > > > > One solution to the above problems is to use per job cluster, > > > but a > > > > > > > > centralized cluster manager can help to manage flink > deployment > > > and > > > > > > jobs > > > > > > > > better. > > > > > > > > > > > > > > > > How you guys think about those? If the community agrees with > > us, > > > we > > > > > > would > > > > > > > > like to propose design and implementation to enhance the > flink > > > > > cluster > > > > > > > > manager. > > > > > > > > -- > > > > > > > > Liu, Renjie > > > > > > > > Software Engineer, MVAD > > > > > > > > -- > > > > > > > > Liu, Renjie > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |