Flip 6 mesos support
Locked
12
12
 
|
Hi all:
I'm reading the source code and it seems that flip6 does not support mesos? According to the design, client send job graph to dispatcher and dispatcher spawn job mananger and resource manager for job execution. But I can't find dispatcher implementation for mesos. -- Liu, Renjie Software Engineer, MVAD |
|
Hi Renjie,
in order to make Mesos work, we only needed to implement a Mesos specific ResourceManager. Look at MesosResourceManager for more details. As dispatcher, we use the StandaloneDispatcher which is spawned by the MesosSessionClusterEntrypoint. Cheers, Till On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> wrote: > Hi all: > I'm reading the source code and it seems that flip6 does not support mesos? > According to the design, client send job graph to dispatcher and dispatcher > spawn job mananger and resource manager for job execution. But I can't find > dispatcher implementation for mesos. > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Hi Till:
Is there any doc on deploying flink in flip6 mode? We want to help testing it. Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > Hi Renjie, > > in order to make Mesos work, we only needed to implement a Mesos specific > ResourceManager. Look at MesosResourceManager for more details. As > dispatcher, we use the StandaloneDispatcher which is spawned by > the MesosSessionClusterEntrypoint. > > Cheers, > Till > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > wrote: > > > Hi all: > > I'm reading the source code and it seems that flip6 does not support > mesos? > > According to the design, client send job graph to dispatcher and > dispatcher > > spawn job mananger and resource manager for job execution. But I can't > find > > dispatcher implementation for mesos. > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
|
Hi Renjie,
in the current master and release-1.5 branch flip-6 is activated by default. If you want to turn it off you have to add `mode: old` to your flink-conf.yaml. I'm really happy that you want to test it out :-) Cheers, Till On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> wrote: > Hi Till: > Is there any doc on deploying flink in flip6 mode? We want to help testing > it. > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > Hi Renjie, > > > > in order to make Mesos work, we only needed to implement a Mesos specific > > ResourceManager. Look at MesosResourceManager for more details. As > > dispatcher, we use the StandaloneDispatcher which is spawned by > > the MesosSessionClusterEntrypoint. > > > > Cheers, > > Till > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > > wrote: > > > > > Hi all: > > > I'm reading the source code and it seems that flip6 does not support > > mesos? > > > According to the design, client send job graph to dispatcher and > > dispatcher > > > spawn job mananger and resource manager for job execution. But I can't > > find > > > dispatcher implementation for mesos. > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Hi Till, have we tested the YARN kerberos integration in flip6? AFAI
remember, YARNSessionFIFOSecuredITCase is not functioning (FLINK-8562 <https://issues.apache.org/jira/browse/FLINK-8562>), do we have similar integration test for flip6? Also, Flink yarn kerberos integration in the old deployment was broken in 1.3 when flip6 is being developed (FLINK-8286 <https://issues.apache.org/jira/browse/FLINK-8286>). Thanks a lot. Shuyi On Wed, Mar 14, 2018 at 9:16 AM, Till Rohrmann <[hidden email]> wrote: > Hi Renjie, > > in the current master and release-1.5 branch flip-6 is activated by > default. If you want to turn it off you have to add `mode: old` to your > flink-conf.yaml. I'm really happy that you want to test it out :-) > > Cheers, > Till > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> > wrote: > > > Hi Till: > > Is there any doc on deploying flink in flip6 mode? We want to help > testing > > it. > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > Hi Renjie, > > > > > > in order to make Mesos work, we only needed to implement a Mesos > specific > > > ResourceManager. Look at MesosResourceManager for more details. As > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > the MesosSessionClusterEntrypoint. > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > > Hi all: > > > > I'm reading the source code and it seems that flip6 does not support > > > mesos? > > > > According to the design, client send job graph to dispatcher and > > > dispatcher > > > > spawn job mananger and resource manager for job execution. But I > can't > > > find > > > > dispatcher implementation for mesos. > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- "So you have to trust that the dots will somehow connect in your future." |
|
|
In reply to this post by Till Rohrmann
Hi, Till:
In fact I'm asking how to deploy other components such as dispatcher, etc. Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > Hi Renjie, > > in the current master and release-1.5 branch flip-6 is activated by > default. If you want to turn it off you have to add `mode: old` to your > flink-conf.yaml. I'm really happy that you want to test it out :-) > > Cheers, > Till > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> > wrote: > > > Hi Till: > > Is there any doc on deploying flink in flip6 mode? We want to help > testing > > it. > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > Hi Renjie, > > > > > > in order to make Mesos work, we only needed to implement a Mesos > specific > > > ResourceManager. Look at MesosResourceManager for more details. As > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > the MesosSessionClusterEntrypoint. > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > > Hi all: > > > > I'm reading the source code and it seems that flip6 does not support > > > mesos? > > > > According to the design, client send job graph to dispatcher and > > > dispatcher > > > > spawn job mananger and resource manager for job execution. But I > can't > > > find > > > > dispatcher implementation for mesos. > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
|
Hi Renjie,
if I am not misunderstand, you just need to start the cluster as normal as before. The dispatcher and resourcemanager are spawned by ClusterEntryPoint (you can have a look at yarn-session.sh & FlinkYarnSessionCli & YarnSessionClusterEntrypoint), and the TM are spawned by ResourceManager lazily (ResourceManager will setup TM according to the submitted job) or spawned by the setup script (you can have a look at start-cluster.sh). Best Regards, Sihua Zhou 发自网易邮箱大师 On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： Hi, Till: In fact I'm asking how to deploy other components such as dispatcher, etc. Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： Hi Renjie, in the current master and release-1.5 branch flip-6 is activated by default. If you want to turn it off you have to add `mode: old` to your flink-conf.yaml. I'm really happy that you want to test it out :-) Cheers, Till On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> wrote: Hi Till: Is there any doc on deploying flink in flip6 mode? We want to help testing it. Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： Hi Renjie, in order to make Mesos work, we only needed to implement a Mesos specific ResourceManager. Look at MesosResourceManager for more details. As dispatcher, we use the StandaloneDispatcher which is spawned by the MesosSessionClusterEntrypoint. Cheers, Till On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> wrote: Hi all: I'm reading the source code and it seems that flip6 does not support mesos? According to the design, client send job graph to dispatcher and dispatcher spawn job mananger and resource manager for job execution. But I can't find dispatcher implementation for mesos. -- Liu, Renjie Software Engineer, MVAD -- Liu, Renjie Software Engineer, MVAD -- Liu, Renjie Software Engineer, MVAD |
|
|
Thanks for the clarification
On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > Hi Renjie, > if I am not misunderstand, you just need to start the cluster as normal as > before. The dispatcher and resourcemanager are spawned by ClusterEntryPoint > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > YarnSessionClusterEntrypoint), and the TM are spawned by ResourceManager > lazily (ResourceManager will setup TM according to the submitted job) or > spawned by the setup script (you can have a look at start-cluster.sh). > > > Best Regards, > Sihua Zhou > > > 发自网易邮箱大师 > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > Hi, Till: > In fact I'm asking how to deploy other components such as dispatcher, etc. > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > Hi Renjie, > > in the current master and release-1.5 branch flip-6 is activated by > default. If you want to turn it off you have to add `mode: old` to your > flink-conf.yaml. I'm really happy that you want to test it out :-) > > Cheers, > Till > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> > wrote: > > Hi Till: > Is there any doc on deploying flink in flip6 mode? We want to help > testing > it. > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > Hi Renjie, > > in order to make Mesos work, we only needed to implement a Mesos > specific > ResourceManager. Look at MesosResourceManager for more details. As > dispatcher, we use the StandaloneDispatcher which is spawned by > the MesosSessionClusterEntrypoint. > > Cheers, > Till > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > wrote: > > Hi all: > I'm reading the source code and it seems that flip6 does not support > mesos? > According to the design, client send job graph to dispatcher and > dispatcher > spawn job mananger and resource manager for job execution. But I > can't > find > dispatcher implementation for mesos. > -- > Liu, Renjie > Software Engineer, MVAD > > > -- > Liu, Renjie > Software Engineer, MVAD > > > -- > Liu, Renjie > Software Engineer, MVAD > Liu, Renjie Software Engineer, MVAD |
|
|
Hi Renjie,
thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're right that we should fix this test. There is FLINK-8562 which seems to address the problem. Will take a look. Additionally, we want to test Kerberos authentication explicitly as part of the release testing for Flink 1.5. I will shortly send around a mail where I will lay out the ongoing testing efforts and where more is needed. Cheers, Till On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu <[hidden email]> wrote: > Thanks for the clarification > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > > > Hi Renjie, > > if I am not misunderstand, you just need to start the cluster as normal > as > > before. The dispatcher and resourcemanager are spawned by > ClusterEntryPoint > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > YarnSessionClusterEntrypoint), and the TM are spawned by ResourceManager > > lazily (ResourceManager will setup TM according to the submitted job) or > > spawned by the setup script (you can have a look at start-cluster.sh). > > > > > > Best Regards, > > Sihua Zhou > > > > > > 发自网易邮箱大师 > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > > Hi, Till: > > In fact I'm asking how to deploy other components such as dispatcher, > etc. > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > > > Hi Renjie, > > > > in the current master and release-1.5 branch flip-6 is activated by > > default. If you want to turn it off you have to add `mode: old` to your > > flink-conf.yaml. I'm really happy that you want to test it out :-) > > > > Cheers, > > Till > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> > > wrote: > > > > Hi Till: > > Is there any doc on deploying flink in flip6 mode? We want to help > > testing > > it. > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > Hi Renjie, > > > > in order to make Mesos work, we only needed to implement a Mesos > > specific > > ResourceManager. Look at MesosResourceManager for more details. As > > dispatcher, we use the StandaloneDispatcher which is spawned by > > the MesosSessionClusterEntrypoint. > > > > Cheers, > > Till > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > > wrote: > > > > Hi all: > > I'm reading the source code and it seems that flip6 does not support > > mesos? > > According to the design, client send job graph to dispatcher and > > dispatcher > > spawn job mananger and resource manager for job execution. But I > > can't > > find > > dispatcher implementation for mesos. > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Hi Till,
This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to resolve the issue, your help to take a look will be great. Please let me know what I can help to test the Kerberos authentication, I am decently familiar with the Kerberos and YARN security part in Flink. As a starting point, I'd suggest to add an integration test similar to YARNSessionFIFOSecuredITCase for flip6. Shuyi On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann <[hidden email]> wrote: > Hi Renjie, > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're right > that we should fix this test. There is FLINK-8562 which seems to address > the problem. Will take a look. > > Additionally, we want to test Kerberos authentication explicitly as part of > the release testing for Flink 1.5. I will shortly send around a mail where > I will lay out the ongoing testing efforts and where more is needed. > > Cheers, > Till > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu <[hidden email]> > wrote: > > > Thanks for the clarification > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > > > > > Hi Renjie, > > > if I am not misunderstand, you just need to start the cluster as normal > > as > > > before. The dispatcher and resourcemanager are spawned by > > ClusterEntryPoint > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > YarnSessionClusterEntrypoint), and the TM are spawned by > ResourceManager > > > lazily (ResourceManager will setup TM according to the submitted job) > or > > > spawned by the setup script (you can have a look at start-cluster.sh). > > > > > > > > > Best Regards, > > > Sihua Zhou > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > > > Hi, Till: > > > In fact I'm asking how to deploy other components such as dispatcher, > > etc. > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > > > > > Hi Renjie, > > > > > > in the current master and release-1.5 branch flip-6 is activated by > > > default. If you want to turn it off you have to add `mode: old` to your > > > flink-conf.yaml. I'm really happy that you want to test it out :-) > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > Hi Till: > > > Is there any doc on deploying flink in flip6 mode? We want to help > > > testing > > > it. > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > > Hi Renjie, > > > > > > in order to make Mesos work, we only needed to implement a Mesos > > > specific > > > ResourceManager. Look at MesosResourceManager for more details. As > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > the MesosSessionClusterEntrypoint. > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > Hi all: > > > I'm reading the source code and it seems that flip6 does not support > > > mesos? > > > According to the design, client send job graph to dispatcher and > > > dispatcher > > > spawn job mananger and resource manager for job execution. But I > > > can't > > > find > > > dispatcher implementation for mesos. > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > -- "So you have to trust that the dots will somehow connect in your future." |
|
|
Hi Shuyi,
thanks for the working on FLINK-8562. Once this issue is fixed, it will automatically be executed on the Flip-6 components. In fact it is already being executed on Flip-6. But what you could help the community with is setting up an automated end-to-end test for the Kerberos integration if you want: https://issues.apache.org/jira/browse/FLINK-8981. The Flink community is currently working on automating more and more tests in order to facilitate faster releases and improve the test coverage. You can find more about this effort here: https://issues.apache.org/jira/browse/FLINK-8970. Cheers, Till On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> wrote: > Hi Till, > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to > resolve the issue, your help to take a look will be great. > > Please let me know what I can help to test the Kerberos authentication, I > am decently familiar with the Kerberos and YARN security part in Flink. > > As a starting point, I'd suggest to add an integration test similar to > YARNSessionFIFOSecuredITCase > for flip6. > > Shuyi > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Renjie, > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're > right > > that we should fix this test. There is FLINK-8562 which seems to address > > the problem. Will take a look. > > > > Additionally, we want to test Kerberos authentication explicitly as part > of > > the release testing for Flink 1.5. I will shortly send around a mail > where > > I will lay out the ongoing testing efforts and where more is needed. > > > > Cheers, > > Till > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu <[hidden email]> > > wrote: > > > > > Thanks for the clarification > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > > > > > > > Hi Renjie, > > > > if I am not misunderstand, you just need to start the cluster as > normal > > > as > > > > before. The dispatcher and resourcemanager are spawned by > > > ClusterEntryPoint > > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > ResourceManager > > > > lazily (ResourceManager will setup TM according to the submitted job) > > or > > > > spawned by the setup script (you can have a look at > start-cluster.sh). > > > > > > > > > > > > Best Regards, > > > > Sihua Zhou > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > > > > Hi, Till: > > > > In fact I'm asking how to deploy other components such as dispatcher, > > > etc. > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > > > > > > > Hi Renjie, > > > > > > > > in the current master and release-1.5 branch flip-6 is activated by > > > > default. If you want to turn it off you have to add `mode: old` to > your > > > > flink-conf.yaml. I'm really happy that you want to test it out :-) > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu <[hidden email] > > > > > > wrote: > > > > > > > > Hi Till: > > > > Is there any doc on deploying flink in flip6 mode? We want to help > > > > testing > > > > it. > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > > > > Hi Renjie, > > > > > > > > in order to make Mesos work, we only needed to implement a Mesos > > > > specific > > > > ResourceManager. Look at MesosResourceManager for more details. As > > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > > the MesosSessionClusterEntrypoint. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu <[hidden email] > > > > > > wrote: > > > > > > > > Hi all: > > > > I'm reading the source code and it seems that flip6 does not support > > > > mesos? > > > > According to the design, client send job graph to dispatcher and > > > > dispatcher > > > > spawn job mananger and resource manager for job execution. But I > > > > can't > > > > find > > > > dispatcher implementation for mesos. > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > > > > -- > "So you have to trust that the dots will somehow connect in your future." > |
|
|
Hi Till,
For FLINK-8562, the test is passing now because it's not really checking the right thing. Yes, I can help with the Kerberos integration ticket. Is there an example on how the e2e test should be structured and invoked? Thanks Shuyi On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann <[hidden email]> wrote: > Hi Shuyi, > > thanks for the working on FLINK-8562. Once this issue is fixed, it will > automatically be executed on the Flip-6 components. In fact it is already > being executed on Flip-6. > > But what you could help the community with is setting up an automated > end-to-end test for the Kerberos integration if you want: > https://issues.apache.org/jira/browse/FLINK-8981. > > The Flink community is currently working on automating more and more tests > in order to facilitate faster releases and improve the test coverage. You > can find more about this effort here: > https://issues.apache.org/jira/browse/FLINK-8970. > > Cheers, > Till > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> wrote: > > > Hi Till, > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to > > resolve the issue, your help to take a look will be great. > > > > Please let me know what I can help to test the Kerberos authentication, I > > am decently familiar with the Kerberos and YARN security part in Flink. > > > > As a starting point, I'd suggest to add an integration test similar to > > YARNSessionFIFOSecuredITCase > > for flip6. > > > > Shuyi > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Renjie, > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're > > right > > > that we should fix this test. There is FLINK-8562 which seems to > address > > > the problem. Will take a look. > > > > > > Additionally, we want to test Kerberos authentication explicitly as > part > > of > > > the release testing for Flink 1.5. I will shortly send around a mail > > where > > > I will lay out the ongoing testing efforts and where more is needed. > > > > > > Cheers, > > > Till > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > > Thanks for the clarification > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > > > > > > > > > Hi Renjie, > > > > > if I am not misunderstand, you just need to start the cluster as > > normal > > > > as > > > > > before. The dispatcher and resourcemanager are spawned by > > > > ClusterEntryPoint > > > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > ResourceManager > > > > > lazily (ResourceManager will setup TM according to the submitted > job) > > > or > > > > > spawned by the setup script (you can have a look at > > start-cluster.sh). > > > > > > > > > > > > > > > Best Regards, > > > > > Sihua Zhou > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > > > > > Hi, Till: > > > > > In fact I'm asking how to deploy other components such as > dispatcher, > > > > etc. > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > > > > > > > > > Hi Renjie, > > > > > > > > > > in the current master and release-1.5 branch flip-6 is activated by > > > > > default. If you want to turn it off you have to add `mode: old` to > > your > > > > > flink-conf.yaml. I'm really happy that you want to test it out :-) > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > Hi Till: > > > > > Is there any doc on deploying flink in flip6 mode? We want to help > > > > > testing > > > > > it. > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > > > > > > Hi Renjie, > > > > > > > > > > in order to make Mesos work, we only needed to implement a Mesos > > > > > specific > > > > > ResourceManager. Look at MesosResourceManager for more details. As > > > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > Hi all: > > > > > I'm reading the source code and it seems that flip6 does not > support > > > > > mesos? > > > > > According to the design, client send job graph to dispatcher and > > > > > dispatcher > > > > > spawn job mananger and resource manager for job execution. But I > > > > > can't > > > > > find > > > > > dispatcher implementation for mesos. > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > "So you have to trust that the dots will somehow connect in your future." > > > -- "So you have to trust that the dots will somehow connect in your future." |
|
|
Hi Shuyi,
best if you look at the other e2e tests in the flink-end-to-end-tests module. For example the Kafka e2e test under flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. Cheers, Till On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email]> wrote: > Hi Till, > > For FLINK-8562, the test is passing now because it's not really > checking the right thing. > > Yes, I can help with the Kerberos integration ticket. > > Is there an example on how the e2e test should be structured and invoked? > > Thanks > Shuyi > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann <[hidden email]> > wrote: > > > Hi Shuyi, > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it will > > automatically be executed on the Flip-6 components. In fact it is already > > being executed on Flip-6. > > > > But what you could help the community with is setting up an automated > > end-to-end test for the Kerberos integration if you want: > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > The Flink community is currently working on automating more and more > tests > > in order to facilitate faster releases and improve the test coverage. You > > can find more about this effort here: > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > Cheers, > > Till > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> wrote: > > > > > Hi Till, > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to > > > resolve the issue, your help to take a look will be great. > > > > > > Please let me know what I can help to test the Kerberos > authentication, I > > > am decently familiar with the Kerberos and YARN security part in Flink. > > > > > > As a starting point, I'd suggest to add an integration test similar to > > > YARNSessionFIFOSecuredITCase > > > for flip6. > > > > > > Shuyi > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Renjie, > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. You're > > > right > > > > that we should fix this test. There is FLINK-8562 which seems to > > address > > > > the problem. Will take a look. > > > > > > > > Additionally, we want to test Kerberos authentication explicitly as > > part > > > of > > > > the release testing for Flink 1.5. I will shortly send around a mail > > > where > > > > I will lay out the ongoing testing efforts and where more is needed. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu <[hidden email] > > > > > > wrote: > > > > > > > > > Thanks for the clarification > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > > > > > > > > > > > Hi Renjie, > > > > > > if I am not misunderstand, you just need to start the cluster as > > > normal > > > > > as > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > ClusterEntryPoint > > > > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > ResourceManager > > > > > > lazily (ResourceManager will setup TM according to the submitted > > job) > > > > or > > > > > > spawned by the setup script (you can have a look at > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > > > > > > Hi, Till: > > > > > > In fact I'm asking how to deploy other components such as > > dispatcher, > > > > > etc. > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is activated > by > > > > > > default. If you want to turn it off you have to add `mode: old` > to > > > your > > > > > > flink-conf.yaml. I'm really happy that you want to test it out > :-) > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > Hi Till: > > > > > > Is there any doc on deploying flink in flip6 mode? We want to > help > > > > > > testing > > > > > > it. > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > in order to make Mesos work, we only needed to implement a Mesos > > > > > > specific > > > > > > ResourceManager. Look at MesosResourceManager for more details. > As > > > > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > Hi all: > > > > > > I'm reading the source code and it seems that flip6 does not > > support > > > > > > mesos? > > > > > > According to the design, client send job graph to dispatcher and > > > > > > dispatcher > > > > > > spawn job mananger and resource manager for job execution. But I > > > > > > can't > > > > > > find > > > > > > dispatcher implementation for mesos. > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > -- > > > "So you have to trust that the dots will somehow connect in your > future." > > > > > > > > > -- > "So you have to trust that the dots will somehow connect in your future." > |
|
|
Hi, Till:
How to specify job manager and task manager resources for different jobs in session mode? On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann <[hidden email]> wrote: > Hi Shuyi, > > best if you look at the other e2e tests in the flink-end-to-end-tests > module. For example the Kafka e2e test under > flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. > > Cheers, > Till > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email]> wrote: > > > Hi Till, > > > > For FLINK-8562, the test is passing now because it's not really > > checking the right thing. > > > > Yes, I can help with the Kerberos integration ticket. > > > > Is there an example on how the e2e test should be structured and invoked? > > > > Thanks > > Shuyi > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Shuyi, > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it will > > > automatically be executed on the Flip-6 components. In fact it is > already > > > being executed on Flip-6. > > > > > > But what you could help the community with is setting up an automated > > > end-to-end test for the Kerberos integration if you want: > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > The Flink community is currently working on automating more and more > > tests > > > in order to facilitate faster releases and improve the test coverage. > You > > > can find more about this effort here: > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > Cheers, > > > Till > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> > wrote: > > > > > > > Hi Till, > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR to > > > > resolve the issue, your help to take a look will be great. > > > > > > > > Please let me know what I can help to test the Kerberos > > authentication, I > > > > am decently familiar with the Kerberos and YARN security part in > Flink. > > > > > > > > As a starting point, I'd suggest to add an integration test similar > to > > > > YARNSessionFIFOSecuredITCase > > > > for flip6. > > > > > > > > Shuyi > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann <[hidden email] > > > > > > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > You're > > > > right > > > > > that we should fix this test. There is FLINK-8562 which seems to > > > address > > > > > the problem. Will take a look. > > > > > > > > > > Additionally, we want to test Kerberos authentication explicitly as > > > part > > > > of > > > > > the release testing for Flink 1.5. I will shortly send around a > > > > where > > > > > I will lay out the ongoing testing efforts and where more is > needed. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> wrote: > > > > > > > > > > > > > Hi Renjie, > > > > > > > if I am not misunderstand, you just need to start the cluster > as > > > > normal > > > > > > as > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > ClusterEntryPoint > > > > > > > (you can have a look at yarn-session.sh & FlinkYarnSessionCli & > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > ResourceManager > > > > > > > lazily (ResourceManager will setup TM according to the > submitted > > > job) > > > > > or > > > > > > > spawned by the setup script (you can have a look at > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> wrote： > > > > > > > Hi, Till: > > > > > > > In fact I'm asking how to deploy other components such as > > > dispatcher, > > > > > > etc. > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 上午12:17写道： > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > activated > > by > > > > > > > default. If you want to turn it off you have to add `mode: old` > > to > > > > your > > > > > > > flink-conf.yaml. I'm really happy that you want to test it out > > :-) > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Till: > > > > > > > Is there any doc on deploying flink in flip6 mode? We want to > > help > > > > > > > testing > > > > > > > it. > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 下午7:08写道： > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > in order to make Mesos work, we only needed to implement a > Mesos > > > > > > > specific > > > > > > > ResourceManager. Look at MesosResourceManager for more details. > > As > > > > > > > dispatcher, we use the StandaloneDispatcher which is spawned by > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi all: > > > > > > > I'm reading the source code and it seems that flip6 does not > > > support > > > > > > > mesos? > > > > > > > According to the design, client send job graph to dispatcher > and > > > > > > > dispatcher > > > > > > > spawn job mananger and resource manager for job execution. But > I > > > > > > > can't > > > > > > > find > > > > > > > dispatcher implementation for mesos. > > > > > > > -- > > > > > > > Liu, Renjie > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Liu, Renjie > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Liu, Renjie > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > "So you have to trust that the dots will somehow connect in your > > future." > > > > > > > > > > > > > > > -- > > "So you have to trust that the dots will somehow connect in your future." > > > Liu, Renjie Software Engineer, MVAD |
|
|
Hi Renjie,
what do you mean with specifying different JM and TM resources for different jobs exactly? Cheers, Till On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu <[hidden email]> wrote: > Hi, Till: > > How to specify job manager and task manager resources for different jobs in > session mode? > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann <[hidden email]> > wrote: > > > Hi Shuyi, > > > > best if you look at the other e2e tests in the flink-end-to-end-tests > > module. For example the Kafka e2e test under > > flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. > > > > Cheers, > > Till > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email]> wrote: > > > > > Hi Till, > > > > > > For FLINK-8562, the test is passing now because it's not really > > > checking the right thing. > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > Is there an example on how the e2e test should be structured and > invoked? > > > > > > Thanks > > > Shuyi > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Shuyi, > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it > will > > > > automatically be executed on the Flip-6 components. In fact it is > > already > > > > being executed on Flip-6. > > > > > > > > But what you could help the community with is setting up an automated > > > > end-to-end test for the Kerberos integration if you want: > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > The Flink community is currently working on automating more and more > > > tests > > > > in order to facilitate faster releases and improve the test coverage. > > You > > > > can find more about this effort here: > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> > > wrote: > > > > > > > > > Hi Till, > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR > to > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > authentication, I > > > > > am decently familiar with the Kerberos and YARN security part in > > Flink. > > > > > > > > > > As a starting point, I'd suggest to add an integration test similar > > to > > > > > YARNSessionFIFOSecuredITCase > > > > > for flip6. > > > > > > > > > > Shuyi > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > > You're > > > > > right > > > > > > that we should fix this test. There is FLINK-8562 which seems to > > > > address > > > > > > the problem. Will take a look. > > > > > > > > > > > > Additionally, we want to test Kerberos authentication explicitly > as > > > > part > > > > > of > > > > > > the release testing for Flink 1.5. I will shortly send around a > > > > > where > > > > > > I will lay out the ongoing testing efforts and where more is > > needed. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> > wrote: > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > if I am not misunderstand, you just need to start the cluster > > as > > > > > normal > > > > > > > as > > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > > ClusterEntryPoint > > > > > > > > (you can have a look at yarn-session.sh & > FlinkYarnSessionCli & > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > > ResourceManager > > > > > > > > lazily (ResourceManager will setup TM according to the > > submitted > > > > job) > > > > > > or > > > > > > > > spawned by the setup script (you can have a look at > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> > wrote： > > > > > > > > Hi, Till: > > > > > > > > In fact I'm asking how to deploy other components such as > > > > dispatcher, > > > > > > > etc. > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 > 上午12:17写道： > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > > activated > > > by > > > > > > > > default. If you want to turn it off you have to add `mode: > old` > > > to > > > > > your > > > > > > > > flink-conf.yaml. I'm really happy that you want to test it > out > > > :-) > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > Is there any doc on deploying flink in flip6 mode? We want to > > > help > > > > > > > > testing > > > > > > > > it. > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 > 下午7:08写道： > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to implement a > > Mesos > > > > > > > > specific > > > > > > > > ResourceManager. Look at MesosResourceManager for more > details. > > > As > > > > > > > > dispatcher, we use the StandaloneDispatcher which is spawned > by > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi all: > > > > > > > > I'm reading the source code and it seems that flip6 does not > > > > support > > > > > > > > mesos? > > > > > > > > According to the design, client send job graph to dispatcher > > and > > > > > > > > dispatcher > > > > > > > > spawn job mananger and resource manager for job execution. > But > > I > > > > > > > > can't > > > > > > > > find > > > > > > > > dispatcher implementation for mesos. > > > > > > > > -- > > > > > > > > Liu, Renjie > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Liu, Renjie > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Liu, Renjie > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > > > > Liu, Renjie > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > "So you have to trust that the dots will somehow connect in your > > > future." > > > > > > > > > > > > > > > > > > > > > -- > > > "So you have to trust that the dots will somehow connect in your > future." > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
For example, we have 2 jobs.
For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 need s10 slots, and I want to deploy these 10 slots in 2 task managers, each with 5 cores and 1G memory. For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 needs 100 slots and I want to deploy these 100 slot in 10 task managers, each with 10 cores and 2G memory. Is this possible? On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <[hidden email]> wrote: > Hi Renjie, > > what do you mean with specifying different JM and TM resources for > different jobs exactly? > > Cheers, > Till > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu <[hidden email]> > wrote: > > > Hi, Till: > > > > How to specify job manager and task manager resources for different jobs > in > > session mode? > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Shuyi, > > > > > > best if you look at the other e2e tests in the flink-end-to-end-tests > > > module. For example the Kafka e2e test under > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_kafka010.sh. > > > > > > Cheers, > > > Till > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email]> > wrote: > > > > > > > Hi Till, > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > checking the right thing. > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > Is there an example on how the e2e test should be structured and > > invoked? > > > > > > > > Thanks > > > > Shuyi > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann <[hidden email] > > > > > > wrote: > > > > > > > > > Hi Shuyi, > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, it > > will > > > > > automatically be executed on the Flip-6 components. In fact it is > > > already > > > > > being executed on Flip-6. > > > > > > > > > > But what you could help the community with is setting up an > automated > > > > > end-to-end test for the Kerberos integration if you want: > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > The Flink community is currently working on automating more and > more > > > > tests > > > > > in order to facilitate faster releases and improve the test > coverage. > > > You > > > > > can find more about this effort here: > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> > > > wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a PR > > to > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > authentication, I > > > > > > am decently familiar with the Kerberos and YARN security part in > > > Flink. > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > similar > > > to > > > > > > YARNSessionFIFOSecuredITCase > > > > > > for flip6. > > > > > > > > > > > > Shuyi > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > > > You're > > > > > > right > > > > > > > that we should fix this test. There is FLINK-8562 which seems > to > > > > > address > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > explicitly > > as > > > > > part > > > > > > of > > > > > > > the release testing for Flink 1.5. I will shortly send around a > > > > > > where > > > > > > > I will lay out the ongoing testing efforts and where more is > > > needed. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> > > wrote: > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > if I am not misunderstand, you just need to start the > cluster > > > as > > > > > > normal > > > > > > > > as > > > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > > > ClusterEntryPoint > > > > > > > > > (you can have a look at yarn-session.sh & > > FlinkYarnSessionCli & > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > > > ResourceManager > > > > > > > > > lazily (ResourceManager will setup TM according to the > > > submitted > > > > > job) > > > > > > > or > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> > > wrote： > > > > > > > > > Hi, Till: > > > > > > > > > In fact I'm asking how to deploy other components such as > > > > > dispatcher, > > > > > > > > etc. > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 > > 上午12:17写道： > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > > > activated > > > > by > > > > > > > > > default. If you want to turn it off you have to add `mode: > > old` > > > > to > > > > > > your > > > > > > > > > flink-conf.yaml. I'm really happy that you want to test it > > out > > > > :-) > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > > Is there any doc on deploying flink in flip6 mode? We want > to > > > > help > > > > > > > > > testing > > > > > > > > > it. > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 > > 下午7:08写道： > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to implement a > > > Mesos > > > > > > > > > specific > > > > > > > > > ResourceManager. Look at MesosResourceManager for more > > details. > > > > As > > > > > > > > > dispatcher, we use the StandaloneDispatcher which is > spawned > > by > > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi all: > > > > > > > > > I'm reading the source code and it seems that flip6 does > not > > > > > support > > > > > > > > > mesos? > > > > > > > > > According to the design, client send job graph to > dispatcher > > > and > > > > > > > > > dispatcher > > > > > > > > > spawn job mananger and resource manager for job execution. > > But > > > I > > > > > > > > > can't > > > > > > > > > find > > > > > > > > > dispatcher implementation for mesos. > > > > > > > > > -- > > > > > > > > > Liu, Renjie > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Liu, Renjie > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Liu, Renjie > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > -- > > > > > > > > Liu, Renjie > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > "So you have to trust that the dots will somehow connect in your > > > > future." > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > "So you have to trust that the dots will somehow connect in your > > future." > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
|
At the moment this is not possible. In order to do this, you will have to
use the per-job mode and run each job on a dedicated Flink cluster. On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu <[hidden email]> wrote: > For example, we have 2 jobs. > For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 > need s10 slots, and I want to deploy these 10 slots in 2 task managers, > each with 5 cores and 1G memory. > > For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 > needs 100 slots and I want to deploy these 100 slot in 10 task managers, > each with 10 cores and 2G memory. > > Is this possible? > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <[hidden email]> > wrote: > > > Hi Renjie, > > > > what do you mean with specifying different JM and TM resources for > > different jobs exactly? > > > > Cheers, > > Till > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu <[hidden email]> > > wrote: > > > > > Hi, Till: > > > > > > How to specify job manager and task manager resources for different > jobs > > in > > > session mode? > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Shuyi, > > > > > > > > best if you look at the other e2e tests in the flink-end-to-end-tests > > > > module. For example the Kafka e2e test under > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > kafka010.sh. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email]> > > wrote: > > > > > > > > > Hi Till, > > > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > > checking the right thing. > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > Is there an example on how the e2e test should be structured and > > > invoked? > > > > > > > > > > Thanks > > > > > Shuyi > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, > it > > > will > > > > > > automatically be executed on the Flip-6 components. In fact it is > > > > already > > > > > > being executed on Flip-6. > > > > > > > > > > > > But what you could help the community with is setting up an > > automated > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > The Flink community is currently working on automating more and > > more > > > > > tests > > > > > > in order to facilitate faster releases and improve the test > > coverage. > > > > You > > > > > > can find more about this effort here: > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen <[hidden email]> > > > > wrote: > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent a > PR > > > to > > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > authentication, I > > > > > > > am decently familiar with the Kerberos and YARN security part > in > > > > Flink. > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > > similar > > > > to > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > for flip6. > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > thanks for the pointer with the YARNSessionFIFOSecuredITCase. > > > > You're > > > > > > > right > > > > > > > > that we should fix this test. There is FLINK-8562 which seems > > to > > > > > > address > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > explicitly > > > as > > > > > > part > > > > > > > of > > > > > > > > the release testing for Flink 1.5. I will shortly send > around a > > > > > > > where > > > > > > > > I will lay out the ongoing testing efforts and where more is > > > > needed. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email]> > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > if I am not misunderstand, you just need to start the > > cluster > > > > as > > > > > > > normal > > > > > > > > > as > > > > > > > > > > before. The dispatcher and resourcemanager are spawned by > > > > > > > > > ClusterEntryPoint > > > > > > > > > > (you can have a look at yarn-session.sh & > > > FlinkYarnSessionCli & > > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned by > > > > > > > > ResourceManager > > > > > > > > > > lazily (ResourceManager will setup TM according to the > > > > submitted > > > > > > job) > > > > > > > > or > > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email]> > > > wrote： > > > > > > > > > > Hi, Till: > > > > > > > > > > In fact I'm asking how to deploy other components such as > > > > > > dispatcher, > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 > > > 上午12:17写道： > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > > > > activated > > > > > by > > > > > > > > > > default. If you want to turn it off you have to add > `mode: > > > old` > > > > > to > > > > > > > your > > > > > > > > > > flink-conf.yaml. I'm really happy that you want to test > it > > > out > > > > > :-) > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > > > Is there any doc on deploying flink in flip6 mode? We > want > > to > > > > > help > > > > > > > > > > testing > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 > > > 下午7:08写道： > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to implement > a > > > > Mesos > > > > > > > > > > specific > > > > > > > > > > ResourceManager. Look at MesosResourceManager for more > > > details. > > > > > As > > > > > > > > > > dispatcher, we use the StandaloneDispatcher which is > > spawned > > > by > > > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi all: > > > > > > > > > > I'm reading the source code and it seems that flip6 does > > not > > > > > > support > > > > > > > > > > mesos? > > > > > > > > > > According to the design, client send job graph to > > dispatcher > > > > and > > > > > > > > > > dispatcher > > > > > > > > > > spawn job mananger and resource manager for job > execution. > > > But > > > > I > > > > > > > > > > can't > > > > > > > > > > find > > > > > > > > > > dispatcher implementation for mesos. > > > > > > > > > > -- > > > > > > > > > > Liu, Renjie > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Liu, Renjie > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Liu, Renjie > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Liu, Renjie > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > "So you have to trust that the dots will somehow connect in > your > > > > > future." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > "So you have to trust that the dots will somehow connect in your > > > future." > > > > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
So how many slots a job manager may consume?
On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann <[hidden email]> wrote: > At the moment this is not possible. In order to do this, you will have to > use the per-job mode and run each job on a dedicated Flink cluster. > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu <[hidden email]> > wrote: > > > For example, we have 2 jobs. > > For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 > > need s10 slots, and I want to deploy these 10 slots in 2 task managers, > > each with 5 cores and 1G memory. > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. Job 2 > > needs 100 slots and I want to deploy these 100 slot in 10 task managers, > > each with 10 cores and 2G memory. > > > > Is this possible? > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <[hidden email]> > > wrote: > > > > > Hi Renjie, > > > > > > what do you mean with specifying different JM and TM resources for > > > different jobs exactly? > > > > > > Cheers, > > > Till > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > > Hi, Till: > > > > > > > > How to specify job manager and task manager resources for different > > jobs > > > in > > > > session mode? > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > Hi Shuyi, > > > > > > > > > > best if you look at the other e2e tests in the > flink-end-to-end-tests > > > > > module. For example the Kafka e2e test under > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > kafka010.sh. > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email]> > > > wrote: > > > > > > > > > > > Hi Till, > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > > > checking the right thing. > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > Is there an example on how the e2e test should be structured and > > > > invoked? > > > > > > > > > > > > Thanks > > > > > > Shuyi > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is fixed, > > it > > > > will > > > > > > > automatically be executed on the Flip-6 components. In fact it > is > > > > > already > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > automated > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > The Flink community is currently working on automating more and > > > more > > > > > > tests > > > > > > > in order to facilitate faster releases and improve the test > > > coverage. > > > > > You > > > > > > > can find more about this effort here: > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > [hidden email]> > > > > > wrote: > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already sent > a > > PR > > > > to > > > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > authentication, I > > > > > > > > am decently familiar with the Kerberos and YARN security part > > in > > > > > Flink. > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > > > similar > > > > > to > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > for flip6. > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > thanks for the pointer with the > YARNSessionFIFOSecuredITCase. > > > > > You're > > > > > > > > right > > > > > > > > > that we should fix this test. There is FLINK-8562 which > seems > > > to > > > > > > > address > > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > > explicitly > > > > as > > > > > > > part > > > > > > > > of > > > > > > > > > the release testing for Flink 1.5. I will shortly send > > around a > > > > > > > > where > > > > > > > > > I will lay out the ongoing testing efforts and where more > is > > > > > needed. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 <[hidden email] > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > if I am not misunderstand, you just need to start the > > > cluster > > > > > as > > > > > > > > normal > > > > > > > > > > as > > > > > > > > > > > before. The dispatcher and resourcemanager are spawned > by > > > > > > > > > > ClusterEntryPoint > > > > > > > > > > > (you can have a look at yarn-session.sh & > > > > FlinkYarnSessionCli & > > > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned > by > > > > > > > > > ResourceManager > > > > > > > > > > > lazily (ResourceManager will setup TM according to the > > > > > submitted > > > > > > > job) > > > > > > > > > or > > > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu<[hidden email] > > > > > > wrote： > > > > > > > > > > > Hi, Till: > > > > > > > > > > > In fact I'm asking how to deploy other components such > as > > > > > > > dispatcher, > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 > > > > 上午12:17写道： > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 is > > > > > activated > > > > > > by > > > > > > > > > > > default. If you want to turn it off you have to add > > `mode: > > > > old` > > > > > > to > > > > > > > > your > > > > > > > > > > > flink-conf.yaml. I'm really happy that you want to test > > it > > > > out > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > > > > Is there any doc on deploying flink in flip6 mode? We > > want > > > to > > > > > > help > > > > > > > > > > > testing > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 > > > > 下午7:08写道： > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to > implement > > a > > > > > Mesos > > > > > > > > > > > specific > > > > > > > > > > > ResourceManager. Look at MesosResourceManager for more > > > > details. > > > > > > As > > > > > > > > > > > dispatcher, we use the StandaloneDispatcher which is > > > spawned > > > > by > > > > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi all: > > > > > > > > > > > I'm reading the source code and it seems that flip6 > does > > > not > > > > > > > support > > > > > > > > > > > mesos? > > > > > > > > > > > According to the design, client send job graph to > > > dispatcher > > > > > and > > > > > > > > > > > dispatcher > > > > > > > > > > > spawn job mananger and resource manager for job > > execution. > > > > But > > > > > I > > > > > > > > > > > can't > > > > > > > > > > > find > > > > > > > > > > > dispatcher implementation for mesos. > > > > > > > > > > > -- > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Liu, Renjie > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > "So you have to trust that the dots will somehow connect in > > your > > > > > > future." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > "So you have to trust that the dots will somehow connect in your > > > > future." > > > > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
|
|
As many as the application needs to run. If you start a job with
parallelism 10 then it will ask for 10 slots (assuming slot sharing). On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu <[hidden email]> wrote: > So how many slots a job manager may consume? > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann <[hidden email]> > wrote: > > > At the moment this is not possible. In order to do this, you will have to > > use the per-job mode and run each job on a dedicated Flink cluster. > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu <[hidden email]> > > wrote: > > > > > For example, we have 2 jobs. > > > For job 1, I want to start job manger with 1 CPU and 100M memory. Job 1 > > > need s10 slots, and I want to deploy these 10 slots in 2 task managers, > > > each with 5 cores and 1G memory. > > > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. Job > 2 > > > needs 100 slots and I want to deploy these 100 slot in 10 task > managers, > > > each with 10 cores and 2G memory. > > > > > > Is this possible? > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > Hi Renjie, > > > > > > > > what do you mean with specifying different JM and TM resources for > > > > different jobs exactly? > > > > > > > > Cheers, > > > > Till > > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu < > [hidden email]> > > > > wrote: > > > > > > > > > Hi, Till: > > > > > > > > > > How to specify job manager and task manager resources for different > > > jobs > > > > in > > > > > session mode? > > > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann < > [hidden email]> > > > > > wrote: > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > best if you look at the other e2e tests in the > > flink-end-to-end-tests > > > > > > module. For example the Kafka e2e test under > > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > > kafka010.sh. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen <[hidden email] > > > > > > wrote: > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not really > > > > > > > checking the right thing. > > > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > > > Is there an example on how the e2e test should be structured > and > > > > > invoked? > > > > > > > > > > > > > > Thanks > > > > > > > Shuyi > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > > [hidden email] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is > fixed, > > > it > > > > > will > > > > > > > > automatically be executed on the Flip-6 components. In fact > it > > is > > > > > > already > > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > > automated > > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > > > The Flink community is currently working on automating more > and > > > > more > > > > > > > tests > > > > > > > > in order to facilitate faster releases and improve the test > > > > coverage. > > > > > > You > > > > > > > > can find more about this effort here: > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Till > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already > sent > > a > > > PR > > > > > to > > > > > > > > > resolve the issue, your help to take a look will be great. > > > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > > authentication, I > > > > > > > > > am decently familiar with the Kerberos and YARN security > part > > > in > > > > > > Flink. > > > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration test > > > > similar > > > > > > to > > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > > for flip6. > > > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > > [hidden email] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > thanks for the pointer with the > > YARNSessionFIFOSecuredITCase. > > > > > > You're > > > > > > > > > right > > > > > > > > > > that we should fix this test. There is FLINK-8562 which > > seems > > > > to > > > > > > > > address > > > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > > > explicitly > > > > > as > > > > > > > > part > > > > > > > > > of > > > > > > > > > > the release testing for Flink 1.5. I will shortly send > > > around a > > > > > > > > > where > > > > > > > > > > I will lay out the ongoing testing efforts and where more > > is > > > > > > needed. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > if I am not misunderstand, you just need to start the > > > > cluster > > > > > > as > > > > > > > > > normal > > > > > > > > > > > as > > > > > > > > > > > > before. The dispatcher and resourcemanager are > spawned > > by > > > > > > > > > > > ClusterEntryPoint > > > > > > > > > > > > (you can have a look at yarn-session.sh & > > > > > FlinkYarnSessionCli & > > > > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are spawned > > by > > > > > > > > > > ResourceManager > > > > > > > > > > > > lazily (ResourceManager will setup TM according to > the > > > > > > submitted > > > > > > > > job) > > > > > > > > > > or > > > > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu< > [hidden email] > > > > > > > > wrote： > > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > In fact I'm asking how to deploy other components > such > > as > > > > > > > > dispatcher, > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月15日周四 > > > > > 上午12:17写道： > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 > is > > > > > > activated > > > > > > > by > > > > > > > > > > > > default. If you want to turn it off you have to add > > > `mode: > > > > > old` > > > > > > > to > > > > > > > > > your > > > > > > > > > > > > flink-conf.yaml. I'm really happy that you want to > test > > > it > > > > > out > > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > > > > > Is there any doc on deploying flink in flip6 mode? We > > > want > > > > to > > > > > > > help > > > > > > > > > > > > testing > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 2018年3月14日周三 > > > > > 下午7:08写道： > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to > > implement > > > a > > > > > > Mesos > > > > > > > > > > > > specific > > > > > > > > > > > > ResourceManager. Look at MesosResourceManager for > more > > > > > details. > > > > > > > As > > > > > > > > > > > > dispatcher, we use the StandaloneDispatcher which is > > > > spawned > > > > > by > > > > > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi all: > > > > > > > > > > > > I'm reading the source code and it seems that flip6 > > does > > > > not > > > > > > > > support > > > > > > > > > > > > mesos? > > > > > > > > > > > > According to the design, client send job graph to > > > > dispatcher > > > > > > and > > > > > > > > > > > > dispatcher > > > > > > > > > > > > spawn job mananger and resource manager for job > > > execution. > > > > > But > > > > > > I > > > > > > > > > > > > can't > > > > > > > > > > > > find > > > > > > > > > > > > dispatcher implementation for mesos. > > > > > > > > > > > > -- > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > "So you have to trust that the dots will somehow connect in > > > your > > > > > > > future." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > "So you have to trust that the dots will somehow connect in > your > > > > > future." > > > > > > > > > > > > > > > > > > -- > > > > > Liu, Renjie > > > > > Software Engineer, MVAD > > > > > > > > > > > > -- > > > Liu, Renjie > > > Software Engineer, MVAD > > > > > > -- > Liu, Renjie > Software Engineer, MVAD > |
|
|
Hi, Till:
In fact, I want to ask the resources consume by job manager Till Rohrmann <[hidden email]> 于 2018年3月21日周三 下午8:17写道： > As many as the application needs to run. If you start a job with > parallelism 10 then it will ask for 10 slots (assuming slot sharing). > > On Wed, Mar 21, 2018 at 12:04 PM, Renjie Liu <[hidden email]> > wrote: > > > So how many slots a job manager may consume? > > > > On Wed, Mar 21, 2018 at 6:50 PM Till Rohrmann <[hidden email]> > > wrote: > > > > > At the moment this is not possible. In order to do this, you will have > to > > > use the per-job mode and run each job on a dedicated Flink cluster. > > > > > > On Wed, Mar 21, 2018 at 11:33 AM, Renjie Liu <[hidden email]> > > > wrote: > > > > > > > For example, we have 2 jobs. > > > > For job 1, I want to start job manger with 1 CPU and 100M memory. > Job 1 > > > > need s10 slots, and I want to deploy these 10 slots in 2 task > managers, > > > > each with 5 cores and 1G memory. > > > > > > > > For job 2, I want to start job manager with 2 CPU and 200M memory. > Job > > 2 > > > > needs 100 slots and I want to deploy these 100 slot in 10 task > > managers, > > > > each with 10 cores and 2G memory. > > > > > > > > Is this possible? > > > > > > > > On Wed, Mar 21, 2018 at 6:19 PM Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > Hi Renjie, > > > > > > > > > > what do you mean with specifying different JM and TM resources for > > > > > different jobs exactly? > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Wed, Mar 21, 2018 at 10:55 AM, Renjie Liu < > > [hidden email]> > > > > > wrote: > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > How to specify job manager and task manager resources for > different > > > > jobs > > > > > in > > > > > > session mode? > > > > > > > > > > > > On Sun, Mar 18, 2018 at 1:10 AM Till Rohrmann < > > [hidden email]> > > > > > > wrote: > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > best if you look at the other e2e tests in the > > > flink-end-to-end-tests > > > > > > > module. For example the Kafka e2e test under > > > > > > > flink/flink-end-to-end-tests/test-scripts/test_streaming_ > > > > kafka010.sh. > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 10:20 PM, Shuyi Chen < > [hidden email] > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > For FLINK-8562, the test is passing now because it's not > really > > > > > > > > checking the right thing. > > > > > > > > > > > > > > > > Yes, I can help with the Kerberos integration ticket. > > > > > > > > > > > > > > > > Is there an example on how the e2e test should be structured > > and > > > > > > invoked? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Shuyi > > > > > > > > > > > > > > > > On Fri, Mar 16, 2018 at 6:51 AM, Till Rohrmann < > > > > [hidden email] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Shuyi, > > > > > > > > > > > > > > > > > > thanks for the working on FLINK-8562. Once this issue is > > fixed, > > > > it > > > > > > will > > > > > > > > > automatically be executed on the Flip-6 components. In fact > > it > > > is > > > > > > > already > > > > > > > > > being executed on Flip-6. > > > > > > > > > > > > > > > > > > But what you could help the community with is setting up an > > > > > automated > > > > > > > > > end-to-end test for the Kerberos integration if you want: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8981. > > > > > > > > > > > > > > > > > > The Flink community is currently working on automating more > > and > > > > > more > > > > > > > > tests > > > > > > > > > in order to facilitate faster releases and improve the test > > > > > coverage. > > > > > > > You > > > > > > > > > can find more about this effort here: > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-8970. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Till > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 8:45 PM, Shuyi Chen < > > > [hidden email]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Till, > > > > > > > > > > > > > > > > > > > > This is Shuyi :) Thanks a lot. In FLINK-8562, I already > > sent > > > a > > > > PR > > > > > > to > > > > > > > > > > resolve the issue, your help to take a look will be > great. > > > > > > > > > > > > > > > > > > > > Please let me know what I can help to test the Kerberos > > > > > > > > authentication, I > > > > > > > > > > am decently familiar with the Kerberos and YARN security > > part > > > > in > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > As a starting point, I'd suggest to add an integration > test > > > > > similar > > > > > > > to > > > > > > > > > > YARNSessionFIFOSecuredITCase > > > > > > > > > > for flip6. > > > > > > > > > > > > > > > > > > > > Shuyi > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:44 AM, Till Rohrmann < > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > thanks for the pointer with the > > > YARNSessionFIFOSecuredITCase. > > > > > > > You're > > > > > > > > > > right > > > > > > > > > > > that we should fix this test. There is FLINK-8562 which > > > seems > > > > > to > > > > > > > > > address > > > > > > > > > > > the problem. Will take a look. > > > > > > > > > > > > > > > > > > > > > > Additionally, we want to test Kerberos authentication > > > > > explicitly > > > > > > as > > > > > > > > > part > > > > > > > > > > of > > > > > > > > > > > the release testing for Flink 1.5. I will shortly send > > > > around a > > > > > > > > > > where > > > > > > > > > > > I will lay out the ongoing testing efforts and where > more > > > is > > > > > > > needed. > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 7:37 AM, Renjie Liu < > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for the clarification > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:30 PM 周思华 < > > [hidden email] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > if I am not misunderstand, you just need to start > the > > > > > cluster > > > > > > > as > > > > > > > > > > normal > > > > > > > > > > > > as > > > > > > > > > > > > > before. The dispatcher and resourcemanager are > > spawned > > > by > > > > > > > > > > > > ClusterEntryPoint > > > > > > > > > > > > > (you can have a look at yarn-session.sh & > > > > > > FlinkYarnSessionCli & > > > > > > > > > > > > > YarnSessionClusterEntrypoint), and the TM are > spawned > > > by > > > > > > > > > > > ResourceManager > > > > > > > > > > > > > lazily (ResourceManager will setup TM according to > > the > > > > > > > submitted > > > > > > > > > job) > > > > > > > > > > > or > > > > > > > > > > > > > spawned by the setup script (you can have a look at > > > > > > > > > > start-cluster.sh). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > Sihua Zhou > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 发自网易邮箱大师 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 03/15/2018 10:14，Renjie Liu< > > [hidden email] > > > > > > > > > > wrote： > > > > > > > > > > > > > Hi, Till: > > > > > > > > > > > > > In fact I'm asking how to deploy other components > > such > > > as > > > > > > > > > dispatcher, > > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 > 2018年3月15日周四 > > > > > > 上午12:17写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > > > > > in the current master and release-1.5 branch flip-6 > > is > > > > > > > activated > > > > > > > > by > > > > > > > > > > > > > default. If you want to turn it off you have to add > > > > `mode: > > > > > > old` > > > > > > > > to > > > > > > > > > > your > > > > > > > > > > > > > flink-conf.yaml. I'm really happy that you want to > > test > > > > it > > > > > > out > > > > > > > > :-) > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 3:03 PM, Renjie Liu < > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Till: > > > > > > > > > > > > > Is there any doc on deploying flink in flip6 mode? > We > > > > want > > > > > to > > > > > > > > help > > > > > > > > > > > > > testing > > > > > > > > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > Till Rohrmann <[hidden email]> 于 > 2018年3月14日周三 > > > > > > 下午7:08写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Renjie, > > > > > > > > > > > > > > > > > > > > > > > > > > in order to make Mesos work, we only needed to > > > implement > > > > a > > > > > > > Mesos > > > > > > > > > > > > > specific > > > > > > > > > > > > > ResourceManager. Look at MesosResourceManager for > > more > > > > > > details. > > > > > > > > As > > > > > > > > > > > > > dispatcher, we use the StandaloneDispatcher which > is > > > > > spawned > > > > > > by > > > > > > > > > > > > > the MesosSessionClusterEntrypoint. > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 14, 2018 at 9:32 AM, Renjie Liu < > > > > > > > > > [hidden email] > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all: > > > > > > > > > > > > > I'm reading the source code and it seems that flip6 > > > does > > > > > not > > > > > > > > > support > > > > > > > > > > > > > mesos? > > > > > > > > > > > > > According to the design, client send job graph to > > > > > dispatcher > > > > > > > and > > > > > > > > > > > > > dispatcher > > > > > > > > > > > > > spawn job mananger and resource manager for job > > > > execution. > > > > > > But > > > > > > > I > > > > > > > > > > > > > can't > > > > > > > > > > > > > find > > > > > > > > > > > > > dispatcher implementation for mesos. > > > > > > > > > > > > > -- > > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Liu, Renjie > > > > > > > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > "So you have to trust that the dots will somehow connect > in > > > > your > > > > > > > > future." > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > "So you have to trust that the dots will somehow connect in > > your > > > > > > future." > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Liu, Renjie > > > > > > Software Engineer, MVAD > > > > > > > > > > > > > > > -- > > > > Liu, Renjie > > > > Software Engineer, MVAD > > > > > > > > > -- > > Liu, Renjie > > Software Engineer, MVAD > > > Liu, Renjie Software Engineer, MVAD |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |