[DISCUSS] Pod template for native K8s integration

[Yang Wang]

Hi devs,

So far we keep developing the native K8s integration for three major
release cycles in Flink.

   -

   Flink 1.10
   -

      Beta version
      -

      Dynamic resource allocation
      -

      Session mode only
      -

   Flink 1.11
   -

      Application mode
      -

      Usability Improvements
      -

      GPU resource support
      -

   Flink 1.12
   -

      Security improvements(kerberos, K8s secret)
      -

      PyFlink support
      -

      K8s HA

Now the native K8s integration is almost production ready. We already have
the common features for deploying a Flink cluster on K8s cluster natively.
However, compared with the current supported features[1] in
flink-on-k8s-operator and the feedback from users(ML and JIRAs), I am
afraid that we still need some advanced features.

   -

   Init containers[2]
   -

   Sidecar containers[3]
   -

   Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount[4]
   -

   Set owner reference for the JobManager deployment[5]
   -

   Support customizing of containers for native kubernetes setup[6]
   -

   Pod Affinity and Anti-affinity
   -

   Pod Security Context
   -

   Pod Environment Variables Reference

Although they may not be used very frequently or we already have some work
around, I think they are still important for the production. So I am
considering whether we could introduce a pod template[7] for all the use
cases. Users could configure a local yaml file for JobManager/TaskManager.
This will let Flink create the JobManager/TaskManager pods with the
template first, not the empty. Of course, the Flink config options for
Kubernetes[8] have high priority and could overwrite these values. The pod
template could help us get rid of maintaining more and more K8s features.


Looking forward to your suggestions.


[1]. https://github.com/GoogleCloudPlatform/flink-on-k8s-operator#features

[2]. https://issues.apache.org/jira/browse/FLINK-15641

[3]. https://issues.apache.org/jira/browse/FLINK-15871

[4]. https://issues.apache.org/jira/browse/FLINK-15649

[5]. https://issues.apache.org/jira/browse/FLINK-20359

[6]. https://issues.apache.org/jira/browse/FLINK-20324

[7]. https://issues.apache.org/jira/browse/FLINK-15656

[8].
https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes


Best,

Yang

[Till Rohrmann]

Hi Yang Wang,

thanks for starting this discussion. I like the idea of providing a general
mechanism to customize Flink's K8s deployment instead of implementing each
feature separately. This should lower the maintenance overhead and people
coming from K8s should already be familiar with how to specify the pod
template.

Cheers,
Till

On Wed, Dec 9, 2020 at 4:44 PM Yang Wang <[hidden email]> wrote:

> Hi devs,
>
> So far we keep developing the native K8s integration for three major
> release cycles in Flink.
>
>    -
>
>    Flink 1.10
>    -
>
>       Beta version
>       -
>
>       Dynamic resource allocation
>       -
>
>       Session mode only
>       -
>
>    Flink 1.11
>    -
>
>       Application mode
>       -
>
>       Usability Improvements
>       -
>
>       GPU resource support
>       -
>
>    Flink 1.12
>    -
>
>       Security improvements(kerberos, K8s secret)
>       -
>
>       PyFlink support
>       -
>
>       K8s HA
>
> Now the native K8s integration is almost production ready. We already have
> the common features for deploying a Flink cluster on K8s cluster natively.
> However, compared with the current supported features[1] in
> flink-on-k8s-operator and the feedback from users(ML and JIRAs), I am
> afraid that we still need some advanced features.
>
>    -
>
>    Init containers[2]
>    -
>
>    Sidecar containers[3]
>    -
>
>    Arbitrary volume(PVC, hostpath, emptyDir, ConfigMap) mount[4]
>    -
>
>    Set owner reference for the JobManager deployment[5]
>    -
>
>    Support customizing of containers for native kubernetes setup[6]
>    -
>
>    Pod Affinity and Anti-affinity
>    -
>
>    Pod Security Context
>    -
>
>    Pod Environment Variables Reference
>
> Although they may not be used very frequently or we already have some work
> around, I think they are still important for the production. So I am
> considering whether we could introduce a pod template[7] for all the use
> cases. Users could configure a local yaml file for JobManager/TaskManager.
> This will let Flink create the JobManager/TaskManager pods with the
> template first, not the empty. Of course, the Flink config options for
> Kubernetes[8] have high priority and could overwrite these values. The pod
> template could help us get rid of maintaining more and more K8s features.
>
>
> Looking forward to your suggestions.
>
>
> [1]. https://github.com/GoogleCloudPlatform/flink-on-k8s-operator#features
>
> [2]. https://issues.apache.org/jira/browse/FLINK-15641
>
> [3]. https://issues.apache.org/jira/browse/FLINK-15871
>
> [4]. https://issues.apache.org/jira/browse/FLINK-15649
>
> [5]. https://issues.apache.org/jira/browse/FLINK-20359
>
> [6]. https://issues.apache.org/jira/browse/FLINK-20324
>
> [7]. https://issues.apache.org/jira/browse/FLINK-15656
>
> [8].
>
> https://ci.apache.org/projects/flink/flink-docs-master/deployment/config.html#kubernetes
>
>
> Best,
>
> Yang
>