[DISCUSS] Active Kubernetes integration phase2

[Yang Wang]

 Hi everyone,


Currently Flink supports the resource management system YARN and Mesos.
However, they were not
designed for fast moving cloud native architectures, and they could not
support mixed workloads (e.g. batch,
streaming, deep learning, web services, etc.) relatively well. At the same
time, Kubernetes is evolving very
fast to fill those gaps and become the de-facto orchestration framework. So
running Flink on Kubernetes is
a very basic requirement for many users.


At least, we have the following advantages when natively running Flink on
Kubernetes.
* Flink KubernetesResourceManager will allocate TaskManager pods
dynamically based on the resource
requirement of the jobs.
* Using Flink bundled scripts to start/stop session cluster on Kuberenetes.
Do not need external tools anymore.
* Compared with Yarn deployment, different Flink clusters could get better
isolation by leveraging the ability
of Kubernetes.


Recently, i also find more and more uses are very interested in running
Flink on Kubernetes natively.
The community has already made some efforts[1] and will be released in
1.10. Welcome to have a taste and
give us your feedback. However, it is a basic requirement and we still need
many features before production.
So i want to start this discussion to collect the requirements that you
have came across. Feel free to share
your valuable thoughts. We will try to conclude and create sub tasks in the
umbrella ticket[2]. Also i will move
some existing tickets there for easier tracking them.


Best,
Yang

[1].
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html
[2]. https://issues.apache.org/jira/browse/FLINK-14460

[Yang Wang]

Hi Canbin Zheng,

I have found that you created some tickets about Flink on Kubernetes. We
just have the same requirements.
Maybe we could have more discussion about the use cases and implementation
details. I have created
a slack channel[1], please join in if you want.

Any dev users if you have some ideas about the new features. Join in us,
let's make Flink on Kubernetes
moving forward together.


[1].
https://slack.com/share/ISUKQ2WG5/cXwlw5HZgp3AE5ElE8m580Pk/enQtOTEyNjcwMDk4NTQ5LWZiNTU0ZmJiZTU4MWU1OTk2YjllNmE0OTg2YjIxYjA2YmE1MWFlOTE4NWFhMDBkMzE4NDQzYjk1YmQwMDI2MzU


Yang Wang <[hidden email]> 于2020年1月19日周日 下午5:07写道：

> Hi everyone,
>
>
> Currently Flink supports the resource management system YARN and Mesos.
> However, they were not
> designed for fast moving cloud native architectures, and they could not
> support mixed workloads (e.g. batch,
> streaming, deep learning, web services, etc.) relatively well. At the same
> time, Kubernetes is evolving very
> fast to fill those gaps and become the de-facto orchestration framework.
> So running Flink on Kubernetes is
> a very basic requirement for many users.
>
>
> At least, we have the following advantages when natively running Flink on
> Kubernetes.
> * Flink KubernetesResourceManager will allocate TaskManager pods
> dynamically based on the resource
> requirement of the jobs.
> * Using Flink bundled scripts to start/stop session cluster on
> Kuberenetes. Do not need external tools anymore.
> * Compared with Yarn deployment, different Flink clusters could get better
> isolation by leveraging the ability
> of Kubernetes.
>
>
> Recently, i also find more and more uses are very interested in running
> Flink on Kubernetes natively.
> The community has already made some efforts[1] and will be released in
> 1.10. Welcome to have a taste and
> give us your feedback. However, it is a basic requirement and we still
> need many features before production.
> So i want to start this discussion to collect the requirements that you
> have came across. Feel free to share
> your valuable thoughts. We will try to conclude and create sub tasks in
> the umbrella ticket[2]. Also i will move
> some existing tickets there for easier tracking them.
>
>
> Best,
> Yang
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html
> [2]. https://issues.apache.org/jira/browse/FLINK-14460
>

[felixzheng]

Hi Yang Wang,

Thanks for your effort on this topic and inviting me, I am glad to join the
future work on native Kubernetes integration, will try to join the slack
channel latter.

Yang Wang <[hidden email]> 于2020年1月19日周日 下午6:04写道：

> Hi Canbin Zheng,
>
> I have found that you created some tickets about Flink on Kubernetes. We
> just have the same requirements.
> Maybe we could have more discussion about the use cases and implementation
> details. I have created
> a slack channel[1], please join in if you want.
>
> Any dev users if you have some ideas about the new features. Join in us,
> let's make Flink on Kubernetes
> moving forward together.
>
>
> [1].
>
> https://slack.com/share/ISUKQ2WG5/cXwlw5HZgp3AE5ElE8m580Pk/enQtOTEyNjcwMDk4NTQ5LWZiNTU0ZmJiZTU4MWU1OTk2YjllNmE0OTg2YjIxYjA2YmE1MWFlOTE4NWFhMDBkMzE4NDQzYjk1YmQwMDI2MzU
>
>
> Yang Wang <[hidden email]> 于2020年1月19日周日 下午5:07写道：
>
> > Hi everyone,
> >
> >
> > Currently Flink supports the resource management system YARN and Mesos.
> > However, they were not
> > designed for fast moving cloud native architectures, and they could not
> > support mixed workloads (e.g. batch,
> > streaming, deep learning, web services, etc.) relatively well. At the
> same
> > time, Kubernetes is evolving very
> > fast to fill those gaps and become the de-facto orchestration framework.
> > So running Flink on Kubernetes is
> > a very basic requirement for many users.
> >
> >
> > At least, we have the following advantages when natively running Flink on
> > Kubernetes.
> > * Flink KubernetesResourceManager will allocate TaskManager pods
> > dynamically based on the resource
> > requirement of the jobs.
> > * Using Flink bundled scripts to start/stop session cluster on
> > Kuberenetes. Do not need external tools anymore.
> > * Compared with Yarn deployment, different Flink clusters could get
> better
> > isolation by leveraging the ability
> > of Kubernetes.
> >
> >
> > Recently, i also find more and more uses are very interested in running
> > Flink on Kubernetes natively.
> > The community has already made some efforts[1] and will be released in
> > 1.10. Welcome to have a taste and
> > give us your feedback. However, it is a basic requirement and we still
> > need many features before production.
> > So i want to start this discussion to collect the requirements that you
> > have came across. Feel free to share
> > your valuable thoughts. We will try to conclude and create sub tasks in
> > the umbrella ticket[2]. Also i will move
> > some existing tickets there for easier tracking them.
> >
> >
> > Best,
> > Yang
> >
> > [1].
> >
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html
> > [2]. https://issues.apache.org/jira/browse/FLINK-14460
> >
>

[Till Rohrmann]

Thanks a lot for driving this discussion Yang Wang. I believe Flink's K8s
integration will become more and more important as more users switch to
K8s. Hence, it is very important that Flink gives users a very K8s
experience and can cover many different use cases.

Cheers,
Till

On Sun, Jan 19, 2020 at 11:46 AM felixzheng zheng <[hidden email]>
wrote:

> Hi Yang Wang,
>
> Thanks for your effort on this topic and inviting me, I am glad to join the
> future work on native Kubernetes integration, will try to join the slack
> channel latter.
>
> Yang Wang <[hidden email]> 于2020年1月19日周日 下午6:04写道：
>
> > Hi Canbin Zheng,
> >
> > I have found that you created some tickets about Flink on Kubernetes. We
> > just have the same requirements.
> > Maybe we could have more discussion about the use cases and
> implementation
> > details. I have created
> > a slack channel[1], please join in if you want.
> >
> > Any dev users if you have some ideas about the new features. Join in us,
> > let's make Flink on Kubernetes
> > moving forward together.
> >
> >
> > [1].
> >
> >
> https://slack.com/share/ISUKQ2WG5/cXwlw5HZgp3AE5ElE8m580Pk/enQtOTEyNjcwMDk4NTQ5LWZiNTU0ZmJiZTU4MWU1OTk2YjllNmE0OTg2YjIxYjA2YmE1MWFlOTE4NWFhMDBkMzE4NDQzYjk1YmQwMDI2MzU
> >
> >
> > Yang Wang <[hidden email]> 于2020年1月19日周日 下午5:07写道：
> >
> > > Hi everyone,
> > >
> > >
> > > Currently Flink supports the resource management system YARN and Mesos.
> > > However, they were not
> > > designed for fast moving cloud native architectures, and they could not
> > > support mixed workloads (e.g. batch,
> > > streaming, deep learning, web services, etc.) relatively well. At the
> > same
> > > time, Kubernetes is evolving very
> > > fast to fill those gaps and become the de-facto orchestration
> framework.
> > > So running Flink on Kubernetes is
> > > a very basic requirement for many users.
> > >
> > >
> > > At least, we have the following advantages when natively running Flink
> on
> > > Kubernetes.
> > > * Flink KubernetesResourceManager will allocate TaskManager pods
> > > dynamically based on the resource
> > > requirement of the jobs.
> > > * Using Flink bundled scripts to start/stop session cluster on
> > > Kuberenetes. Do not need external tools anymore.
> > > * Compared with Yarn deployment, different Flink clusters could get
> > better
> > > isolation by leveraging the ability
> > > of Kubernetes.
> > >
> > >
> > > Recently, i also find more and more uses are very interested in running
> > > Flink on Kubernetes natively.
> > > The community has already made some efforts[1] and will be released in
> > > 1.10. Welcome to have a taste and
> > > give us your feedback. However, it is a basic requirement and we still
> > > need many features before production.
> > > So i want to start this discussion to collect the requirements that you
> > > have came across. Feel free to share
> > > your valuable thoughts. We will try to conclude and create sub tasks in
> > > the umbrella ticket[2]. Also i will move
> > > some existing tickets there for easier tracking them.
> > >
> > >
> > > Best,
> > > Yang
> > >
> > > [1].
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/native_kubernetes.html
> > > [2]. https://issues.apache.org/jira/browse/FLINK-14460
> > >
> >
>