Flink operators for Kubernetes

[Anand Swaminathan]

Hello All,

This is Anand from Lyft. Just wanted to send out a note that we at Lyft are
working on building Operators (https://coreos.com/operators/) to support
deploying, managing Flink Applications in Kubernetes. The operator is
responsible for creating Flink Clusters, start jobs and seamlessly
transition a job from one cluster to another during deployment.

We hope to open source it at some point once deployed and tested within
Lyft. Let us know if any of you are looking for something like this. We
will be happy to collaborate.

Thanks,
Anand
(Flyte @Lyft)

[isunjin]

Sounds interesting.

 

Currently Flink can run in standalone mode in a kubernets cluster, details see here.   There is also a JIRA Flink-9953: Active kubernetes integration is target for run Flink on Kubernetes natively.

 

Jin

 

On 10/2/18, 1:30 PM, "Anand Swaminathan" <[hidden email]> wrote:

 

    Hello All,

   

    This is Anand from Lyft. Just wanted to send out a note that we at Lyft are

    working on building Operators (https://coreos.com/operators/) to support

    deploying, managing Flink Applications in Kubernetes. The operator is

    responsible for creating Flink Clusters, start jobs and seamlessly

    transition a job from one cluster to another during deployment.

   

    We hope to open source it at some point once deployed and tested within

    Lyft. Let us know if any of you are looking for something like this. We

    will be happy to collaborate.

   

    Thanks,

    Anand

    (Flyte @Lyft)

   

[Till Rohrmann]

Great to hear that you intend to open source your K8s operators. I would be
keen to see what and how you do things with the operator. If there are
things to change on the Flink side in order to improve the integration,
then let's discuss them.

Cheers,
Till

On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:

> Sounds interesting.
>
>
>
> Currently Flink can run in standalone mode in a kubernets cluster, details
> see here.   There is also a JIRA Flink-9953: Active kubernetes integration
> is target for run Flink on Kubernetes natively.
>
>
>
> Jin
>
>
>
> On 10/2/18, 1:30 PM, "Anand Swaminathan" <[hidden email]>
> wrote:
>
>
>
>     Hello All,
>
>
>
>     This is Anand from Lyft. Just wanted to send out a note that we at
> Lyft are
>
>     working on building Operators (https://coreos.com/operators/) to
> support
>
>     deploying, managing Flink Applications in Kubernetes. The operator is
>
>     responsible for creating Flink Clusters, start jobs and seamlessly
>
>     transition a job from one cluster to another during deployment.
>
>
>
>     We hope to open source it at some point once deployed and tested within
>
>     Lyft. Let us know if any of you are looking for something like this. We
>
>     will be happy to collaborate.
>
>
>
>     Thanks,
>
>     Anand
>
>     (Flyte @Lyft)
>
>
>
>

[Anand Swaminathan]

For sure.

I am targeting to have the Operator for applications running Flink version
1.6 and above. From an integration standpoint, the ability to control the
Flink Application from outside the cluster using APIs is a huge deal. So
any improvement on that area is a huge value addition.

Thanks,
Anand

On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann <[hidden email]> wrote:

> Great to hear that you intend to open source your K8s operators. I would be
> keen to see what and how you do things with the operator. If there are
> things to change on the Flink side in order to improve the integration,
> then let's discuss them.
>
> Cheers,
> Till
>
> On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:
>
> > Sounds interesting.
> >
> >
> >
> > Currently Flink can run in standalone mode in a kubernets cluster,
> details
> > see here.   There is also a JIRA Flink-9953: Active kubernetes
> integration
> > is target for run Flink on Kubernetes natively.
> >
> >
> >
> > Jin
> >
> >
> >
> > On 10/2/18, 1:30 PM, "Anand Swaminathan" <[hidden email]
> >
> > wrote:
> >
> >
> >
> >     Hello All,
> >
> >
> >
> >     This is Anand from Lyft. Just wanted to send out a note that we at
> > Lyft are
> >
> >     working on building Operators (https://coreos.com/operators/) to
> > support
> >
> >     deploying, managing Flink Applications in Kubernetes. The operator is
> >
> >     responsible for creating Flink Clusters, start jobs and seamlessly
> >
> >     transition a job from one cluster to another during deployment.
> >
> >
> >
> >     We hope to open source it at some point once deployed and tested
> within
> >
> >     Lyft. Let us know if any of you are looking for something like this.
> We
> >
> >     will be happy to collaborate.
> >
> >
> >
> >     Thanks,
> >
> >     Anand
> >
> >     (Flyte @Lyft)
> >
> >
> >
> >
>

[Anand Swaminathan]

Hello,

I have documented our work of running Flink applications on Kubernetes
using FlinkK8sOperator - here
<https://docs.google.com/document/d/1_AITTq0fay-NwUIqNfqN0pum-erOmS_4kYZnlMxw6_o/edit#heading=h.ge413lh374xj>

We have finished the implementation of the FlinkK8sOperator, and are
currently testing it with few applications. Please let us know your
feedback.

Thanks,
Anand
(Flyte @Lyft)


On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann <[hidden email]> wrote:

> Great to hear that you intend to open source your K8s operators. I would be
> keen to see what and how you do things with the operator. If there are
> things to change on the Flink side in order to improve the integration,
> then let's discuss them.
>
> Cheers,
> Till
>
> On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:
>
> > Sounds interesting.
> >
> >
> >
> > Currently Flink can run in standalone mode in a kubernets cluster,
> details
> > see here.   There is also a JIRA Flink-9953: Active kubernetes
> integration
> > is target for run Flink on Kubernetes natively.
> >
> >
> >
> > Jin
> >
> >
> >
> > On 10/2/18, 1:30 PM, "Anand Swaminathan" <[hidden email]
> >
> > wrote:
> >
> >
> >
> >     Hello All,
> >
> >
> >
> >     This is Anand from Lyft. Just wanted to send out a note that we at
> > Lyft are
> >
> >     working on building Operators (https://coreos.com/operators/) to
> > support
> >
> >     deploying, managing Flink Applications in Kubernetes. The operator is
> >
> >     responsible for creating Flink Clusters, start jobs and seamlessly
> >
> >     transition a job from one cluster to another during deployment.
> >
> >
> >
> >     We hope to open source it at some point once deployed and tested
> within
> >
> >     Lyft. Let us know if any of you are looking for something like this.
> We
> >
> >     will be happy to collaborate.
> >
> >
> >
> >     Thanks,
> >
> >     Anand
> >
> >     (Flyte @Lyft)
> >
> >
> >
> >
>

[Till Rohrmann]

Thanks for sharing your work with the community.

Cheers,
Till

On Fri, Nov 2, 2018 at 1:16 AM Anand Swaminathan
<[hidden email]> wrote:

> Hello,
>
> I have documented our work of running Flink applications on Kubernetes
> using FlinkK8sOperator - here
> <
> https://docs.google.com/document/d/1_AITTq0fay-NwUIqNfqN0pum-erOmS_4kYZnlMxw6_o/edit#heading=h.ge413lh374xj
> >
>
> We have finished the implementation of the FlinkK8sOperator, and are
> currently testing it with few applications. Please let us know your
> feedback.
>
> Thanks,
> Anand
> (Flyte @Lyft)
>
>
> On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann <[hidden email]> wrote:
>
> > Great to hear that you intend to open source your K8s operators. I would
> be
> > keen to see what and how you do things with the operator. If there are
> > things to change on the Flink side in order to improve the integration,
> > then let's discuss them.
> >
> > Cheers,
> > Till
> >
> > On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:
> >
> > > Sounds interesting.
> > >
> > >
> > >
> > > Currently Flink can run in standalone mode in a kubernets cluster,
> > details
> > > see here.   There is also a JIRA Flink-9953: Active kubernetes
> > integration
> > > is target for run Flink on Kubernetes natively.
> > >
> > >
> > >
> > > Jin
> > >
> > >
> > >
> > > On 10/2/18, 1:30 PM, "Anand Swaminathan"
> <[hidden email]
> > >
> > > wrote:
> > >
> > >
> > >
> > >     Hello All,
> > >
> > >
> > >
> > >     This is Anand from Lyft. Just wanted to send out a note that we at
> > > Lyft are
> > >
> > >     working on building Operators (https://coreos.com/operators/) to
> > > support
> > >
> > >     deploying, managing Flink Applications in Kubernetes. The operator
> is
> > >
> > >     responsible for creating Flink Clusters, start jobs and seamlessly
> > >
> > >     transition a job from one cluster to another during deployment.
> > >
> > >
> > >
> > >     We hope to open source it at some point once deployed and tested
> > within
> > >
> > >     Lyft. Let us know if any of you are looking for something like
> this.
> > We
> > >
> > >     will be happy to collaborate.
> > >
> > >
> > >
> > >     Thanks,
> > >
> > >     Anand
> > >
> > >     (Flyte @Lyft)
> > >
> > >
> > >
> > >
> >
>

[Thomas Weise]

In case anyone comes across this thread in the archives,
the FlinkK8sOperator is now available here:

https://github.com/lyft/flinkk8soperator

The community is invited to check it out, provide feedback (use github
issues) or even better, join and contribute to it.

Thomas


On Fri, Nov 2, 2018 at 2:08 AM Till Rohrmann <[hidden email]> wrote:

> Thanks for sharing your work with the community.
>
> Cheers,
> Till
>
> On Fri, Nov 2, 2018 at 1:16 AM Anand Swaminathan
> <[hidden email]> wrote:
>
> > Hello,
> >
> > I have documented our work of running Flink applications on Kubernetes
> > using FlinkK8sOperator - here
> > <
> >
> https://docs.google.com/document/d/1_AITTq0fay-NwUIqNfqN0pum-erOmS_4kYZnlMxw6_o/edit#heading=h.ge413lh374xj
> > >
> >
> > We have finished the implementation of the FlinkK8sOperator, and are
> > currently testing it with few applications. Please let us know your
> > feedback.
> >
> > Thanks,
> > Anand
> > (Flyte @Lyft)
> >
> >
> > On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann <[hidden email]>
> wrote:
> >
> > > Great to hear that you intend to open source your K8s operators. I
> would
> > be
> > > keen to see what and how you do things with the operator. If there are
> > > things to change on the Flink side in order to improve the integration,
> > > then let's discuss them.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:
> > >
> > > > Sounds interesting.
> > > >
> > > >
> > > >
> > > > Currently Flink can run in standalone mode in a kubernets cluster,
> > > details
> > > > see here.   There is also a JIRA Flink-9953: Active kubernetes
> > > integration
> > > > is target for run Flink on Kubernetes natively.
> > > >
> > > >
> > > >
> > > > Jin
> > > >
> > > >
> > > >
> > > > On 10/2/18, 1:30 PM, "Anand Swaminathan"
> > <[hidden email]
> > > >
> > > > wrote:
> > > >
> > > >
> > > >
> > > >     Hello All,
> > > >
> > > >
> > > >
> > > >     This is Anand from Lyft. Just wanted to send out a note that we
> at
> > > > Lyft are
> > > >
> > > >     working on building Operators (https://coreos.com/operators/) to
> > > > support
> > > >
> > > >     deploying, managing Flink Applications in Kubernetes. The
> operator
> > is
> > > >
> > > >     responsible for creating Flink Clusters, start jobs and
> seamlessly
> > > >
> > > >     transition a job from one cluster to another during deployment.
> > > >
> > > >
> > > >
> > > >     We hope to open source it at some point once deployed and tested
> > > within
> > > >
> > > >     Lyft. Let us know if any of you are looking for something like
> > this.
> > > We
> > > >
> > > >     will be happy to collaborate.
> > > >
> > > >
> > > >
> > > >     Thanks,
> > > >
> > > >     Anand
> > > >
> > > >     (Flyte @Lyft)
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

[Anand Swaminathan]

Hey guys,

We open sourced the flink on K8s operator -
https://github.com/lyft/flinkk8soperator and have been stabilizing it for a
while. Several folks from the community have already been using it.
<https://github.com/lyft/flinkk8soperator>

Please check it out and give us your feedback. If you have any questions,
feel free to join our FlinkK8sOperator slack
<https://join.slack.com/t/flinkk8soperator/shared_invite/enQtNzIxMjc5NDYxODkxLTEwMThmN2I0M2QwYjM3ZDljYTFhMGRiNDUzM2FjZGYzNTRjYWNmYTE1NzNlNWM2YWM5NzNiNGFhMTkxZjA4OGU>
channel
as well.

Thanks,
Anand

On Thu, Oct 4, 2018 at 12:46 PM Anand Swaminathan <[hidden email]>
wrote:

> For sure.
>
> I am targeting to have the Operator for applications running Flink version
> 1.6 and above. From an integration standpoint, the ability to control the
> Flink Application from outside the cluster using APIs is a huge deal. So
> any improvement on that area is a huge value addition.
>
> Thanks,
> Anand
>
> On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann <[hidden email]> wrote:
>
>> Great to hear that you intend to open source your K8s operators. I would
>> be
>> keen to see what and how you do things with the operator. If there are
>> things to change on the Flink side in order to improve the integration,
>> then let's discuss them.
>>
>> Cheers,
>> Till
>>
>> On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:
>>
>> > Sounds interesting.
>> >
>> >
>> >
>> > Currently Flink can run in standalone mode in a kubernets cluster,
>> details
>> > see here.   There is also a JIRA Flink-9953: Active kubernetes
>> integration
>> > is target for run Flink on Kubernetes natively.
>> >
>> >
>> >
>> > Jin
>> >
>> >
>> >
>> > On 10/2/18, 1:30 PM, "Anand Swaminathan" <[hidden email]
>> >
>> > wrote:
>> >
>> >
>> >
>> >     Hello All,
>> >
>> >
>> >
>> >     This is Anand from Lyft. Just wanted to send out a note that we at
>> > Lyft are
>> >
>> >     working on building Operators (https://coreos.com/operators/) to
>> > support
>> >
>> >     deploying, managing Flink Applications in Kubernetes. The operator
>> is
>> >
>> >     responsible for creating Flink Clusters, start jobs and seamlessly
>> >
>> >     transition a job from one cluster to another during deployment.
>> >
>> >
>> >
>> >     We hope to open source it at some point once deployed and tested
>> within
>> >
>> >     Lyft. Let us know if any of you are looking for something like
>> this. We
>> >
>> >     will be happy to collaborate.
>> >
>> >
>> >
>> >     Thanks,
>> >
>> >     Anand
>> >
>> >     (Flyte @Lyft)
>> >
>> >
>> >
>> >
>>
>

[Till Rohrmann]

Thanks a lot for letting the community know Anand. Great to see that
Flink's open source ecosystem is growing. Have you reached out to Robert or
Becket in order to include the flinkk8soperator project on Flink's
ecosystem website [1]?

I've cross posted this thread to the user ML so that Flink users will learn
about it as well.

[1] https://flink.apache.org/ecosystem.html

Cheers,
Till

On Fri, Sep 13, 2019 at 11:11 PM Anand Swaminathan
<[hidden email]> wrote:

> Hey guys,
>
> We open sourced the flink on K8s operator -
> https://github.com/lyft/flinkk8soperator and have been stabilizing it for
> a
> while. Several folks from the community have already been using it.
> <https://github.com/lyft/flinkk8soperator>
>
> Please check it out and give us your feedback. If you have any questions,
> feel free to join our FlinkK8sOperator slack
> <
> https://join.slack.com/t/flinkk8soperator/shared_invite/enQtNzIxMjc5NDYxODkxLTEwMThmN2I0M2QwYjM3ZDljYTFhMGRiNDUzM2FjZGYzNTRjYWNmYTE1NzNlNWM2YWM5NzNiNGFhMTkxZjA4OGU
> >
> channel
> as well.
>
> Thanks,
> Anand
>
> On Thu, Oct 4, 2018 at 12:46 PM Anand Swaminathan <[hidden email]>
> wrote:
>
> > For sure.
> >
> > I am targeting to have the Operator for applications running Flink
> version
> > 1.6 and above. From an integration standpoint, the ability to control the
> > Flink Application from outside the cluster using APIs is a huge deal. So
> > any improvement on that area is a huge value addition.
> >
> > Thanks,
> > Anand
> >
> > On Thu, Oct 4, 2018 at 1:19 AM Till Rohrmann <[hidden email]>
> wrote:
> >
> >> Great to hear that you intend to open source your K8s operators. I would
> >> be
> >> keen to see what and how you do things with the operator. If there are
> >> things to change on the Flink side in order to improve the integration,
> >> then let's discuss them.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Wed, Oct 3, 2018 at 2:52 AM Jin Sun <[hidden email]> wrote:
> >>
> >> > Sounds interesting.
> >> >
> >> >
> >> >
> >> > Currently Flink can run in standalone mode in a kubernets cluster,
> >> details
> >> > see here.   There is also a JIRA Flink-9953: Active kubernetes
> >> integration
> >> > is target for run Flink on Kubernetes natively.
> >> >
> >> >
> >> >
> >> > Jin
> >> >
> >> >
> >> >
> >> > On 10/2/18, 1:30 PM, "Anand Swaminathan"
> <[hidden email]
> >> >
> >> > wrote:
> >> >
> >> >
> >> >
> >> >     Hello All,
> >> >
> >> >
> >> >
> >> >     This is Anand from Lyft. Just wanted to send out a note that we at
> >> > Lyft are
> >> >
> >> >     working on building Operators (https://coreos.com/operators/) to
> >> > support
> >> >
> >> >     deploying, managing Flink Applications in Kubernetes. The operator
> >> is
> >> >
> >> >     responsible for creating Flink Clusters, start jobs and seamlessly
> >> >
> >> >     transition a job from one cluster to another during deployment.
> >> >
> >> >
> >> >
> >> >     We hope to open source it at some point once deployed and tested
> >> within
> >> >
> >> >     Lyft. Let us know if any of you are looking for something like
> >> this. We
> >> >
> >> >     will be happy to collaborate.
> >> >
> >> >
> >> >
> >> >     Thanks,
> >> >
> >> >     Anand
> >> >
> >> >     (Flyte @Lyft)
> >> >
> >> >
> >> >
> >> >
> >>
> >
>