FLIP-6 / Kubernetes

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

FLIP-6 / Kubernetes

Thomas Weise
Hi,

What would be a good starting point to try out Flink on Kubernetes (any
examples / tutorials)?

Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?

Thanks,
Thomas
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Till Rohrmann
Hi Thomas,

I think the one way to get started would be to adapt the Flink docker
images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6
components.

Flink 1.5 won't come with a dedicated integration with Kubernetes which is
able to start new pods. However, it should work that you manually or by the
virtues of an external system start new pods which can be used. Once the
new pods have been started and the TaskExecutors have registered one would
have to rescale the job manually to the new parallelism.

[1] https://flink.apache.org/news/2017/05/16/official-docker-image.html
[2] https://hub.docker.com/r/_/flink/

Cheers,
Till

On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]> wrote:

> Hi,
>
> What would be a good starting point to try out Flink on Kubernetes (any
> examples / tutorials)?
>
> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?
>
> Thanks,
> Thomas
>
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Till Rohrmann
I forgot to add Flink's K8 documentation [1] which might also be helpful
with getting started.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html

Cheers,
Till

On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <[hidden email]>
wrote:

> Hi Thomas,
>
> I think the one way to get started would be to adapt the Flink docker
> images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6
> components.
>
> Flink 1.5 won't come with a dedicated integration with Kubernetes which is
> able to start new pods. However, it should work that you manually or by the
> virtues of an external system start new pods which can be used. Once the
> new pods have been started and the TaskExecutors have registered one would
> have to rescale the job manually to the new parallelism.
>
> [1] https://flink.apache.org/news/2017/05/16/official-docker-image.html
> [2] https://hub.docker.com/r/_/flink/
>
> Cheers,
> Till
>
> On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]> wrote:
>
>> Hi,
>>
>> What would be a good starting point to try out Flink on Kubernetes (any
>> examples / tutorials)?
>>
>> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?
>>
>> Thanks,
>> Thomas
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Eron Wright-2
 Till, is it possible to package a Flink application as a self-contained
deployment on Kubernetes?  I mean, run a Flink application using 'flink
run' such that it launches its own RM/JM and waits for a sufficient # of
TMs to join?

Thanks!

On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <[hidden email]> wrote:

> I forgot to add Flink's K8 documentation [1] which might also be helpful
> with getting started.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/
> kubernetes.html
>
> Cheers,
> Till
>
> On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi Thomas,
> >
> > I think the one way to get started would be to adapt the Flink docker
> > images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6
> > components.
> >
> > Flink 1.5 won't come with a dedicated integration with Kubernetes which
> is
> > able to start new pods. However, it should work that you manually or by
> the
> > virtues of an external system start new pods which can be used. Once the
> > new pods have been started and the TaskExecutors have registered one
> would
> > have to rescale the job manually to the new parallelism.
> >
> > [1] https://flink.apache.org/news/2017/05/16/official-docker-image.html
> > [2] https://hub.docker.com/r/_/flink/
> >
> > Cheers,
> > Till
> >
> > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]> wrote:
> >
> >> Hi,
> >>
> >> What would be a good starting point to try out Flink on Kubernetes (any
> >> examples / tutorials)?
> >>
> >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?
> >>
> >> Thanks,
> >> Thomas
> >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Stephan Ewen
@Eron That is definitely the way we want to suggest as the way to use k8s
in the future. This feature did not make it for 1.5, but should come very
soon after.

@Thomas An implementation of a ResourceManager for k8s should come in the
near future. Would be happy to jump on a joint FLIP, after the 1.5 release
and Flink Forward (in three weeks or so).

On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright <[hidden email]> wrote:

>  Till, is it possible to package a Flink application as a self-contained
> deployment on Kubernetes?  I mean, run a Flink application using 'flink
> run' such that it launches its own RM/JM and waits for a sufficient # of
> TMs to join?
>
> Thanks!
>
> On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <[hidden email]>
> wrote:
>
> > I forgot to add Flink's K8 documentation [1] which might also be helpful
> > with getting started.
> >
> > [1]
> > https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/
> > kubernetes.html
> >
> > Cheers,
> > Till
> >
> > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > I think the one way to get started would be to adapt the Flink docker
> > > images [1,2] to run with Flink 1.5. Per default, they will use the
> Flip-6
> > > components.
> > >
> > > Flink 1.5 won't come with a dedicated integration with Kubernetes which
> > is
> > > able to start new pods. However, it should work that you manually or by
> > the
> > > virtues of an external system start new pods which can be used. Once
> the
> > > new pods have been started and the TaskExecutors have registered one
> > would
> > > have to rescale the job manually to the new parallelism.
> > >
> > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> image.html
> > > [2] https://hub.docker.com/r/_/flink/
> > >
> > > Cheers,
> > > Till
> > >
> > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]> wrote:
> > >
> > >> Hi,
> > >>
> > >> What would be a good starting point to try out Flink on Kubernetes
> (any
> > >> examples / tutorials)?
> > >>
> > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> Kubernetes?
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Christophe Jolif
Hi Stephan,

> This feature did not make it for 1.5, but should come very
soon after.

Are you talking about a 1.5.1 here? What is the envisioned timeline?

Thanks!


On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen <[hidden email]> wrote:

> @Eron That is definitely the way we want to suggest as the way to use k8s
> in the future. This feature did not make it for 1.5, but should come very
> soon after.
>
> @Thomas An implementation of a ResourceManager for k8s should come in the
> near future. Would be happy to jump on a joint FLIP, after the 1.5 release
> and Flink Forward (in three weeks or so).
>
> On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright <[hidden email]> wrote:
>
> >  Till, is it possible to package a Flink application as a self-contained
> > deployment on Kubernetes?  I mean, run a Flink application using 'flink
> > run' such that it launches its own RM/JM and waits for a sufficient # of
> > TMs to join?
> >
> > Thanks!
> >
> > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > I forgot to add Flink's K8 documentation [1] which might also be
> helpful
> > > with getting started.
> > >
> > > [1]
> > > https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/
> > > kubernetes.html
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <[hidden email]>
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > I think the one way to get started would be to adapt the Flink docker
> > > > images [1,2] to run with Flink 1.5. Per default, they will use the
> > Flip-6
> > > > components.
> > > >
> > > > Flink 1.5 won't come with a dedicated integration with Kubernetes
> which
> > > is
> > > > able to start new pods. However, it should work that you manually or
> by
> > > the
> > > > virtues of an external system start new pods which can be used. Once
> > the
> > > > new pods have been started and the TaskExecutors have registered one
> > > would
> > > > have to rescale the job manually to the new parallelism.
> > > >
> > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > image.html
> > > > [2] https://hub.docker.com/r/_/flink/
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]>
> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> What would be a good starting point to try out Flink on Kubernetes
> > (any
> > > >> examples / tutorials)?
> > > >>
> > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > Kubernetes?
> > > >>
> > > >> Thanks,
> > > >> Thomas
> > > >>
> > > >
> > > >
> > >
> >
>



--
Christophe
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Stephan Ewen
Hi Christophe!

I could imagine 1.5.1 or 1.6.

It is important to point out that you can use Flink with Kubernetes quite
well already. That addition is mainly intended to make it more idiomatic
for Kubernetes  users.

Best,
Stephan


On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif <[hidden email]> wrote:

> Hi Stephan,
>
> > This feature did not make it for 1.5, but should come very
> soon after.
>
> Are you talking about a 1.5.1 here? What is the envisioned timeline?
>
> Thanks!
>
>
> On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen <[hidden email]> wrote:
>
> > @Eron That is definitely the way we want to suggest as the way to use k8s
> > in the future. This feature did not make it for 1.5, but should come very
> > soon after.
> >
> > @Thomas An implementation of a ResourceManager for k8s should come in the
> > near future. Would be happy to jump on a joint FLIP, after the 1.5
> release
> > and Flink Forward (in three weeks or so).
> >
> > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright <[hidden email]>
> wrote:
> >
> > >  Till, is it possible to package a Flink application as a
> self-contained
> > > deployment on Kubernetes?  I mean, run a Flink application using 'flink
> > > run' such that it launches its own RM/JM and waits for a sufficient #
> of
> > > TMs to join?
> > >
> > > Thanks!
> > >
> > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <[hidden email]>
> > > wrote:
> > >
> > > > I forgot to add Flink's K8 documentation [1] which might also be
> > helpful
> > > > with getting started.
> > > >
> > > > [1]
> > > > https://ci.apache.org/projects/flink/flink-docs-
> master/ops/deployment/
> > > > kubernetes.html
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > I think the one way to get started would be to adapt the Flink
> docker
> > > > > images [1,2] to run with Flink 1.5. Per default, they will use the
> > > Flip-6
> > > > > components.
> > > > >
> > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes
> > which
> > > > is
> > > > > able to start new pods. However, it should work that you manually
> or
> > by
> > > > the
> > > > > virtues of an external system start new pods which can be used.
> Once
> > > the
> > > > > new pods have been started and the TaskExecutors have registered
> one
> > > > would
> > > > > have to rescale the job manually to the new parallelism.
> > > > >
> > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > > image.html
> > > > > [2] https://hub.docker.com/r/_/flink/
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]>
> > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> What would be a good starting point to try out Flink on Kubernetes
> > > (any
> > > > >> examples / tutorials)?
> > > > >>
> > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > > Kubernetes?
> > > > >>
> > > > >> Thanks,
> > > > >> Thomas
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Christophe
>
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Eron Wright-2
Thanks Stephan for the information, it's exciting.   This is an aspect for
which the FLIP-6 design doc is lacking detail; somewhere we should work
through how this will work (e.g. from a job recovery and upgrade
perspective).   If there's a doc or an open issue, please share, thanks!

On Wed, Mar 21, 2018 at 5:41 AM, Stephan Ewen <[hidden email]> wrote:

> Hi Christophe!
>
> I could imagine 1.5.1 or 1.6.
>
> It is important to point out that you can use Flink with Kubernetes quite
> well already. That addition is mainly intended to make it more idiomatic
> for Kubernetes  users.
>
> Best,
> Stephan
>
>
> On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif <[hidden email]>
> wrote:
>
> > Hi Stephan,
> >
> > > This feature did not make it for 1.5, but should come very
> > soon after.
> >
> > Are you talking about a 1.5.1 here? What is the envisioned timeline?
> >
> > Thanks!
> >
> >
> > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen <[hidden email]> wrote:
> >
> > > @Eron That is definitely the way we want to suggest as the way to use
> k8s
> > > in the future. This feature did not make it for 1.5, but should come
> very
> > > soon after.
> > >
> > > @Thomas An implementation of a ResourceManager for k8s should come in
> the
> > > near future. Would be happy to jump on a joint FLIP, after the 1.5
> > release
> > > and Flink Forward (in three weeks or so).
> > >
> > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright <[hidden email]>
> > wrote:
> > >
> > > >  Till, is it possible to package a Flink application as a
> > self-contained
> > > > deployment on Kubernetes?  I mean, run a Flink application using
> 'flink
> > > > run' such that it launches its own RM/JM and waits for a sufficient #
> > of
> > > > TMs to join?
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > I forgot to add Flink's K8 documentation [1] which might also be
> > > helpful
> > > > > with getting started.
> > > > >
> > > > > [1]
> > > > > https://ci.apache.org/projects/flink/flink-docs-
> > master/ops/deployment/
> > > > > kubernetes.html
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hi Thomas,
> > > > > >
> > > > > > I think the one way to get started would be to adapt the Flink
> > docker
> > > > > > images [1,2] to run with Flink 1.5. Per default, they will use
> the
> > > > Flip-6
> > > > > > components.
> > > > > >
> > > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes
> > > which
> > > > > is
> > > > > > able to start new pods. However, it should work that you manually
> > or
> > > by
> > > > > the
> > > > > > virtues of an external system start new pods which can be used.
> > Once
> > > > the
> > > > > > new pods have been started and the TaskExecutors have registered
> > one
> > > > > would
> > > > > > have to rescale the job manually to the new parallelism.
> > > > > >
> > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > > > image.html
> > > > > > [2] https://hub.docker.com/r/_/flink/
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]>
> > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> What would be a good starting point to try out Flink on
> Kubernetes
> > > > (any
> > > > > >> examples / tutorials)?
> > > > > >>
> > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > > > Kubernetes?
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Thomas
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Christophe
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: FLIP-6 / Kubernetes

Thomas Weise
Thanks everyone for the info.

I have used the Flink docker containers to launch a Flink cluster, but what
I'm looking forward to is more along the lines asked by Eron: submitting a
Flink pipeline to a k8s cluster without having to deal with the traditional
Flink cluster. I think this would be very valuable, especially if the
resources can be adjusted dynamically.

Looking forward to discuss more around Flink Forward!

Cheers,
Thomas


On Wed, Mar 21, 2018 at 8:02 AM, Eron Wright <[hidden email]> wrote:

> Thanks Stephan for the information, it's exciting.   This is an aspect for
> which the FLIP-6 design doc is lacking detail; somewhere we should work
> through how this will work (e.g. from a job recovery and upgrade
> perspective).   If there's a doc or an open issue, please share, thanks!
>
> On Wed, Mar 21, 2018 at 5:41 AM, Stephan Ewen <[hidden email]> wrote:
>
> > Hi Christophe!
> >
> > I could imagine 1.5.1 or 1.6.
> >
> > It is important to point out that you can use Flink with Kubernetes quite
> > well already. That addition is mainly intended to make it more idiomatic
> > for Kubernetes  users.
> >
> > Best,
> > Stephan
> >
> >
> > On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif <[hidden email]>
> > wrote:
> >
> > > Hi Stephan,
> > >
> > > > This feature did not make it for 1.5, but should come very
> > > soon after.
> > >
> > > Are you talking about a 1.5.1 here? What is the envisioned timeline?
> > >
> > > Thanks!
> > >
> > >
> > > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen <[hidden email]>
> wrote:
> > >
> > > > @Eron That is definitely the way we want to suggest as the way to use
> > k8s
> > > > in the future. This feature did not make it for 1.5, but should come
> > very
> > > > soon after.
> > > >
> > > > @Thomas An implementation of a ResourceManager for k8s should come in
> > the
> > > > near future. Would be happy to jump on a joint FLIP, after the 1.5
> > > release
> > > > and Flink Forward (in three weeks or so).
> > > >
> > > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright <[hidden email]>
> > > wrote:
> > > >
> > > > >  Till, is it possible to package a Flink application as a
> > > self-contained
> > > > > deployment on Kubernetes?  I mean, run a Flink application using
> > 'flink
> > > > > run' such that it launches its own RM/JM and waits for a
> sufficient #
> > > of
> > > > > TMs to join?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <
> [hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > > > I forgot to add Flink's K8 documentation [1] which might also be
> > > > helpful
> > > > > > with getting started.
> > > > > >
> > > > > > [1]
> > > > > > https://ci.apache.org/projects/flink/flink-docs-
> > > master/ops/deployment/
> > > > > > kubernetes.html
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Thomas,
> > > > > > >
> > > > > > > I think the one way to get started would be to adapt the Flink
> > > docker
> > > > > > > images [1,2] to run with Flink 1.5. Per default, they will use
> > the
> > > > > Flip-6
> > > > > > > components.
> > > > > > >
> > > > > > > Flink 1.5 won't come with a dedicated integration with
> Kubernetes
> > > > which
> > > > > > is
> > > > > > > able to start new pods. However, it should work that you
> manually
> > > or
> > > > by
> > > > > > the
> > > > > > > virtues of an external system start new pods which can be used.
> > > Once
> > > > > the
> > > > > > > new pods have been started and the TaskExecutors have
> registered
> > > one
> > > > > > would
> > > > > > > have to rescale the job manually to the new parallelism.
> > > > > > >
> > > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > > > > image.html
> > > > > > > [2] https://hub.docker.com/r/_/flink/
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise <[hidden email]>
> > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> What would be a good starting point to try out Flink on
> > Kubernetes
> > > > > (any
> > > > > > >> examples / tutorials)?
> > > > > > >>
> > > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > > > > Kubernetes?
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Thomas
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Christophe
> > >
> >
>