support launch more than one pipeline in a yarn session

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

support launch more than one pipeline in a yarn session

chenqin
Hi there,

We are researching launch more than one pipeline in a flink yarn session.
It's sort of like cluster mode where user pick off multiple pipelines on a
job manager. Is there any plan to support this use case? If not, why not?
Is it because of better isolation per pipeline or some implementation
insights? What would be recommended way to add this part if some one plan
to work on it?

Speaking of our use case, we want to allow runs arbitrary numbers of
pipeline in a given flink yarn session so that people don't need to chase
rabbits and manage lots of web dashboards at same time.

Thanks,
Chen
Reply | Threaded
Open this post in threaded view
|

Re: support launch more than one pipeline in a yarn session

Till Rohrmann
Hi Chen,

I'm not sure what you mean with a pipeline but Flink supports the
submission of multiple jobs to the same cluster (in standalone as well as
Yarn session mode). You simply have to make sure that there are enough
slots for all jobs to be executed at the same time.

Cheers,
Till
On Jun 28, 2016 6:47 AM, "Chen Qin" <[hidden email]> wrote:

> Hi there,
>
> We are researching launch more than one pipeline in a flink yarn session.
> It's sort of like cluster mode where user pick off multiple pipelines on a
> job manager. Is there any plan to support this use case? If not, why not?
> Is it because of better isolation per pipeline or some implementation
> insights? What would be recommended way to add this part if some one plan
> to work on it?
>
> Speaking of our use case, we want to allow runs arbitrary numbers of
> pipeline in a given flink yarn session so that people don't need to chase
> rabbits and manage lots of web dashboards at same time.
>
> Thanks,
> Chen
>