[DISCUSS] Service Authorization (redux)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Service Authorization (redux)

Eron Wright-2
Hello, now might be a good time to revisit an important enhancement to
Flink security, so-called service authorization.   This means the hardening
of a Flink cluster against unauthorized use with some sort of
authentication and authorization scheme.   Today, Flink relies entirely on
network isolation to protect itself from unauthorized job submission and
control, and to protect the secrets contained within a Flink cluster.
This is a problem in multi-user environments like YARN/Mesos/K8.

Last fall, an effort was made to implement service authorization but the PR
was ultimately rejected.   The idea was to add a simple secret key to all
network communication between the client, JM, and TM.   Akka itself has
such a feature which formed the basis of the solution.  There are usability
challenges with this solution, including a dependency on SSL.

Since then, the situation has evolved somewhat, and the use of SSL mutual
authentication is more viable.   Mutual auth is supported in Akka 2.4.12+
(or could be backported to Flakka).  My proposal is:

1. Upgrade Akka or backport the functionality to Flakka (see commit
5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
2. Implement SSL on any endpoint that doesn't yet support it (e.g.
queryable state).
3. Enable mutual auth in Akka and implement it on non-Akka endpoints.
4. Implement a simple authorization layer that accepts any authenticated
connection.
5. (stretch) generate and store a certificate automatically in YARN mode.
6. (stretch) Develop an alternate authentication method for the Web UI.

Are folks interested in this capability?  Thoughts on the use of SSL mutual
auth versus something else?  Thanks!

-Eron
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Aljoscha Krettek-2
Hi Eron,

I think after Dropping support for Java 7 we will move to Akka 2.4+, so we should be good there. I think quite some users should find a (more) secure Flink interesting.

Best,
Aljoscha

> On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]> wrote:
>
> Hello, now might be a good time to revisit an important enhancement to
> Flink security, so-called service authorization.   This means the hardening
> of a Flink cluster against unauthorized use with some sort of
> authentication and authorization scheme.   Today, Flink relies entirely on
> network isolation to protect itself from unauthorized job submission and
> control, and to protect the secrets contained within a Flink cluster.
> This is a problem in multi-user environments like YARN/Mesos/K8.
>
> Last fall, an effort was made to implement service authorization but the PR
> was ultimately rejected.   The idea was to add a simple secret key to all
> network communication between the client, JM, and TM.   Akka itself has
> such a feature which formed the basis of the solution.  There are usability
> challenges with this solution, including a dependency on SSL.
>
> Since then, the situation has evolved somewhat, and the use of SSL mutual
> authentication is more viable.   Mutual auth is supported in Akka 2.4.12+
> (or could be backported to Flakka).  My proposal is:
>
> 1. Upgrade Akka or backport the functionality to Flakka (see commit
> 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
> 2. Implement SSL on any endpoint that doesn't yet support it (e.g.
> queryable state).
> 3. Enable mutual auth in Akka and implement it on non-Akka endpoints.
> 4. Implement a simple authorization layer that accepts any authenticated
> connection.
> 5. (stretch) generate and store a certificate automatically in YARN mode.
> 6. (stretch) Develop an alternate authentication method for the Web UI.
>
> Are folks interested in this capability?  Thoughts on the use of SSL mutual
> auth versus something else?  Thanks!
>
> -Eron

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Till Rohrmann
Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we need
at least Scala 2.11.

Cheers,
Till

On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <[hidden email]>
wrote:

> Hi Eron,
>
> I think after Dropping support for Java 7 we will move to Akka 2.4+, so we
> should be good there. I think quite some users should find a (more) secure
> Flink interesting.
>
> Best,
> Aljoscha
> > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]> wrote:
> >
> > Hello, now might be a good time to revisit an important enhancement to
> > Flink security, so-called service authorization.   This means the
> hardening
> > of a Flink cluster against unauthorized use with some sort of
> > authentication and authorization scheme.   Today, Flink relies entirely
> on
> > network isolation to protect itself from unauthorized job submission and
> > control, and to protect the secrets contained within a Flink cluster.
> > This is a problem in multi-user environments like YARN/Mesos/K8.
> >
> > Last fall, an effort was made to implement service authorization but the
> PR
> > was ultimately rejected.   The idea was to add a simple secret key to all
> > network communication between the client, JM, and TM.   Akka itself has
> > such a feature which formed the basis of the solution.  There are
> usability
> > challenges with this solution, including a dependency on SSL.
> >
> > Since then, the situation has evolved somewhat, and the use of SSL mutual
> > authentication is more viable.   Mutual auth is supported in Akka 2.4.12+
> > (or could be backported to Flakka).  My proposal is:
> >
> > 1. Upgrade Akka or backport the functionality to Flakka (see commit
> > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
> > 2. Implement SSL on any endpoint that doesn't yet support it (e.g.
> > queryable state).
> > 3. Enable mutual auth in Akka and implement it on non-Akka endpoints.
> > 4. Implement a simple authorization layer that accepts any authenticated
> > connection.
> > 5. (stretch) generate and store a certificate automatically in YARN mode.
> > 6. (stretch) Develop an alternate authentication method for the Web UI.
> >
> > Are folks interested in this capability?  Thoughts on the use of SSL
> mutual
> > auth versus something else?  Thanks!
> >
> > -Eron
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Eron Wright-2
Thanks Till and Aljoscha for the feedback.

Seems there are two ways to proceed here, if we accept mutual SSL as the
basis.

a) Backport mutual-auth support from Akka 2.4 to Flakka.
b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4 (FLINK-3662).

Let's assume (a) for now.



On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <[hidden email]> wrote:

> Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we need
> at least Scala 2.11.
>
> Cheers,
> Till
>
> On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > Hi Eron,
> >
> > I think after Dropping support for Java 7 we will move to Akka 2.4+, so
> we
> > should be good there. I think quite some users should find a (more)
> secure
> > Flink interesting.
> >
> > Best,
> > Aljoscha
> > > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]> wrote:
> > >
> > > Hello, now might be a good time to revisit an important enhancement to
> > > Flink security, so-called service authorization.   This means the
> > hardening
> > > of a Flink cluster against unauthorized use with some sort of
> > > authentication and authorization scheme.   Today, Flink relies entirely
> > on
> > > network isolation to protect itself from unauthorized job submission
> and
> > > control, and to protect the secrets contained within a Flink cluster.
> > > This is a problem in multi-user environments like YARN/Mesos/K8.
> > >
> > > Last fall, an effort was made to implement service authorization but
> the
> > PR
> > > was ultimately rejected.   The idea was to add a simple secret key to
> all
> > > network communication between the client, JM, and TM.   Akka itself has
> > > such a feature which formed the basis of the solution.  There are
> > usability
> > > challenges with this solution, including a dependency on SSL.
> > >
> > > Since then, the situation has evolved somewhat, and the use of SSL
> mutual
> > > authentication is more viable.   Mutual auth is supported in Akka
> 2.4.12+
> > > (or could be backported to Flakka).  My proposal is:
> > >
> > > 1. Upgrade Akka or backport the functionality to Flakka (see commit
> > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
> > > 2. Implement SSL on any endpoint that doesn't yet support it (e.g.
> > > queryable state).
> > > 3. Enable mutual auth in Akka and implement it on non-Akka endpoints.
> > > 4. Implement a simple authorization layer that accepts any
> authenticated
> > > connection.
> > > 5. (stretch) generate and store a certificate automatically in YARN
> mode.
> > > 6. (stretch) Develop an alternate authentication method for the Web UI.
> > >
> > > Are folks interested in this capability?  Thoughts on the use of SSL
> > mutual
> > > auth versus something else?  Thanks!
> > >
> > > -Eron
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Till Rohrmann
Alternatively there would also be an option

c) only support mutual auth for Akka 2.4+ if the backport is unrealistic to
do

But this of course would break security for Scala 2.10. On the other hand
people are already using Flink without this feature.

Cheers,
Till

On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright <[hidden email]> wrote:

> Thanks Till and Aljoscha for the feedback.
>
> Seems there are two ways to proceed here, if we accept mutual SSL as the
> basis.
>
> a) Backport mutual-auth support from Akka 2.4 to Flakka.
> b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4 (FLINK-3662).
>
> Let's assume (a) for now.
>
>
>
> On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <[hidden email]>
> wrote:
>
> > Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we
> need
> > at least Scala 2.11.
> >
> > Cheers,
> > Till
> >
> > On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> > > Hi Eron,
> > >
> > > I think after Dropping support for Java 7 we will move to Akka 2.4+, so
> > we
> > > should be good there. I think quite some users should find a (more)
> > secure
> > > Flink interesting.
> > >
> > > Best,
> > > Aljoscha
> > > > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]> wrote:
> > > >
> > > > Hello, now might be a good time to revisit an important enhancement
> to
> > > > Flink security, so-called service authorization.   This means the
> > > hardening
> > > > of a Flink cluster against unauthorized use with some sort of
> > > > authentication and authorization scheme.   Today, Flink relies
> entirely
> > > on
> > > > network isolation to protect itself from unauthorized job submission
> > and
> > > > control, and to protect the secrets contained within a Flink cluster.
> > > > This is a problem in multi-user environments like YARN/Mesos/K8.
> > > >
> > > > Last fall, an effort was made to implement service authorization but
> > the
> > > PR
> > > > was ultimately rejected.   The idea was to add a simple secret key to
> > all
> > > > network communication between the client, JM, and TM.   Akka itself
> has
> > > > such a feature which formed the basis of the solution.  There are
> > > usability
> > > > challenges with this solution, including a dependency on SSL.
> > > >
> > > > Since then, the situation has evolved somewhat, and the use of SSL
> > mutual
> > > > authentication is more viable.   Mutual auth is supported in Akka
> > 2.4.12+
> > > > (or could be backported to Flakka).  My proposal is:
> > > >
> > > > 1. Upgrade Akka or backport the functionality to Flakka (see commit
> > > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
> > > > 2. Implement SSL on any endpoint that doesn't yet support it (e.g.
> > > > queryable state).
> > > > 3. Enable mutual auth in Akka and implement it on non-Akka endpoints.
> > > > 4. Implement a simple authorization layer that accepts any
> > authenticated
> > > > connection.
> > > > 5. (stretch) generate and store a certificate automatically in YARN
> > mode.
> > > > 6. (stretch) Develop an alternate authentication method for the Web
> UI.
> > > >
> > > > Are folks interested in this capability?  Thoughts on the use of SSL
> > > mutual
> > > > auth versus something else?  Thanks!
> > > >
> > > > -Eron
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Ufuk Celebi-2
I haven't followed this discussion in detail nor am I familiar with
the service authorization topic or Flakka, but a) sounds like a lot of
maintenance work to me.

If possible I would go with c) and maybe start a discussion about
dropping Scala 2.10 support to check whether that is a viable option
or not.

– Ufuk


On Thu, Aug 3, 2017 at 1:59 PM, Till Rohrmann <[hidden email]> wrote:

> Alternatively there would also be an option
>
> c) only support mutual auth for Akka 2.4+ if the backport is unrealistic to
> do
>
> But this of course would break security for Scala 2.10. On the other hand
> people are already using Flink without this feature.
>
> Cheers,
> Till
>
> On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright <[hidden email]> wrote:
>
>> Thanks Till and Aljoscha for the feedback.
>>
>> Seems there are two ways to proceed here, if we accept mutual SSL as the
>> basis.
>>
>> a) Backport mutual-auth support from Akka 2.4 to Flakka.
>> b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4 (FLINK-3662).
>>
>> Let's assume (a) for now.
>>
>>
>>
>> On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <[hidden email]>
>> wrote:
>>
>> > Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we
>> need
>> > at least Scala 2.11.
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <[hidden email]>
>> > wrote:
>> >
>> > > Hi Eron,
>> > >
>> > > I think after Dropping support for Java 7 we will move to Akka 2.4+, so
>> > we
>> > > should be good there. I think quite some users should find a (more)
>> > secure
>> > > Flink interesting.
>> > >
>> > > Best,
>> > > Aljoscha
>> > > > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]> wrote:
>> > > >
>> > > > Hello, now might be a good time to revisit an important enhancement
>> to
>> > > > Flink security, so-called service authorization.   This means the
>> > > hardening
>> > > > of a Flink cluster against unauthorized use with some sort of
>> > > > authentication and authorization scheme.   Today, Flink relies
>> entirely
>> > > on
>> > > > network isolation to protect itself from unauthorized job submission
>> > and
>> > > > control, and to protect the secrets contained within a Flink cluster.
>> > > > This is a problem in multi-user environments like YARN/Mesos/K8.
>> > > >
>> > > > Last fall, an effort was made to implement service authorization but
>> > the
>> > > PR
>> > > > was ultimately rejected.   The idea was to add a simple secret key to
>> > all
>> > > > network communication between the client, JM, and TM.   Akka itself
>> has
>> > > > such a feature which formed the basis of the solution.  There are
>> > > usability
>> > > > challenges with this solution, including a dependency on SSL.
>> > > >
>> > > > Since then, the situation has evolved somewhat, and the use of SSL
>> > mutual
>> > > > authentication is more viable.   Mutual auth is supported in Akka
>> > 2.4.12+
>> > > > (or could be backported to Flakka).  My proposal is:
>> > > >
>> > > > 1. Upgrade Akka or backport the functionality to Flakka (see commit
>> > > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
>> > > > 2. Implement SSL on any endpoint that doesn't yet support it (e.g.
>> > > > queryable state).
>> > > > 3. Enable mutual auth in Akka and implement it on non-Akka endpoints.
>> > > > 4. Implement a simple authorization layer that accepts any
>> > authenticated
>> > > > connection.
>> > > > 5. (stretch) generate and store a certificate automatically in YARN
>> > mode.
>> > > > 6. (stretch) Develop an alternate authentication method for the Web
>> UI.
>> > > >
>> > > > Are folks interested in this capability?  Thoughts on the use of SSL
>> > > mutual
>> > > > auth versus something else?  Thanks!
>> > > >
>> > > > -Eron
>> > >
>> > >
>> >
>>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Eron Wright-2
Till, with (c) are you suggesting that we'd use Akka 2.3 for Scala 2.10 and
Akka 2.4+ for Scala 2.11+?   Sounds reasonable but I don't know how
feasible it is.   I will say I'm optimistic because a) Akka 2.4 is said to
be binary compatible, and b) the Flakka fork appears to be subsumed by 2.4.

Let us then take (c) as the tentative plan.

I agree the community should discuss dropping Scala 2.10 but I don't want
to drive that conversation.

Thanks


On Thu, Aug 3, 2017 at 6:24 AM, Ufuk Celebi <[hidden email]> wrote:

> I haven't followed this discussion in detail nor am I familiar with
> the service authorization topic or Flakka, but a) sounds like a lot of
> maintenance work to me.
>
> If possible I would go with c) and maybe start a discussion about
> dropping Scala 2.10 support to check whether that is a viable option
> or not.
>
> – Ufuk
>
>
> On Thu, Aug 3, 2017 at 1:59 PM, Till Rohrmann <[hidden email]>
> wrote:
> > Alternatively there would also be an option
> >
> > c) only support mutual auth for Akka 2.4+ if the backport is unrealistic
> to
> > do
> >
> > But this of course would break security for Scala 2.10. On the other hand
> > people are already using Flink without this feature.
> >
> > Cheers,
> > Till
> >
> > On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright <[hidden email]>
> wrote:
> >
> >> Thanks Till and Aljoscha for the feedback.
> >>
> >> Seems there are two ways to proceed here, if we accept mutual SSL as the
> >> basis.
> >>
> >> a) Backport mutual-auth support from Akka 2.4 to Flakka.
> >> b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4 (FLINK-3662).
> >>
> >> Let's assume (a) for now.
> >>
> >>
> >>
> >> On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <[hidden email]>
> >> wrote:
> >>
> >> > Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we
> >> need
> >> > at least Scala 2.11.
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <[hidden email]
> >
> >> > wrote:
> >> >
> >> > > Hi Eron,
> >> > >
> >> > > I think after Dropping support for Java 7 we will move to Akka
> 2.4+, so
> >> > we
> >> > > should be good there. I think quite some users should find a (more)
> >> > secure
> >> > > Flink interesting.
> >> > >
> >> > > Best,
> >> > > Aljoscha
> >> > > > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]>
> wrote:
> >> > > >
> >> > > > Hello, now might be a good time to revisit an important
> enhancement
> >> to
> >> > > > Flink security, so-called service authorization.   This means the
> >> > > hardening
> >> > > > of a Flink cluster against unauthorized use with some sort of
> >> > > > authentication and authorization scheme.   Today, Flink relies
> >> entirely
> >> > > on
> >> > > > network isolation to protect itself from unauthorized job
> submission
> >> > and
> >> > > > control, and to protect the secrets contained within a Flink
> cluster.
> >> > > > This is a problem in multi-user environments like YARN/Mesos/K8.
> >> > > >
> >> > > > Last fall, an effort was made to implement service authorization
> but
> >> > the
> >> > > PR
> >> > > > was ultimately rejected.   The idea was to add a simple secret
> key to
> >> > all
> >> > > > network communication between the client, JM, and TM.   Akka
> itself
> >> has
> >> > > > such a feature which formed the basis of the solution.  There are
> >> > > usability
> >> > > > challenges with this solution, including a dependency on SSL.
> >> > > >
> >> > > > Since then, the situation has evolved somewhat, and the use of SSL
> >> > mutual
> >> > > > authentication is more viable.   Mutual auth is supported in Akka
> >> > 2.4.12+
> >> > > > (or could be backported to Flakka).  My proposal is:
> >> > > >
> >> > > > 1. Upgrade Akka or backport the functionality to Flakka (see
> commit
> >> > > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
> >> > > > 2. Implement SSL on any endpoint that doesn't yet support it (e.g.
> >> > > > queryable state).
> >> > > > 3. Enable mutual auth in Akka and implement it on non-Akka
> endpoints.
> >> > > > 4. Implement a simple authorization layer that accepts any
> >> > authenticated
> >> > > > connection.
> >> > > > 5. (stretch) generate and store a certificate automatically in
> YARN
> >> > mode.
> >> > > > 6. (stretch) Develop an alternate authentication method for the
> Web
> >> UI.
> >> > > >
> >> > > > Are folks interested in this capability?  Thoughts on the use of
> SSL
> >> > > mutual
> >> > > > auth versus something else?  Thanks!
> >> > > >
> >> > > > -Eron
> >> > >
> >> > >
> >> >
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Eron Wright-2
Hi folks, I'm happy to share with you a draft of a FLIP for service
authorization.   As I mentioned at the top of this thread, the goal is to
protect a deployed Flink cluster/session from unauthorized use.   In the
doc, I propose the use of SSL client authentication for internal
communication, plus YARN/Kubernetes/Mesos-specific functionality to achieve
single sign-on for the web interface/API.

FLIP-? - Service Authorization
<https://docs.google.com/document/d/13IRPb2GdL842rIzMgEn0ibOQHNku6W8aMf1p7gCPJjg/edit?usp=sharing>

Feel free to make comments and to give feedback on whether this feature
would be useful to you.

Thanks!
Eron


On Thu, Aug 3, 2017 at 11:11 AM, Eron Wright <[hidden email]> wrote:

> Till, with (c) are you suggesting that we'd use Akka 2.3 for Scala 2.10
> and Akka 2.4+ for Scala 2.11+?   Sounds reasonable but I don't know how
> feasible it is.   I will say I'm optimistic because a) Akka 2.4 is said to
> be binary compatible, and b) the Flakka fork appears to be subsumed by 2.4.
>
> Let us then take (c) as the tentative plan.
>
> I agree the community should discuss dropping Scala 2.10 but I don't want
> to drive that conversation.
>
> Thanks
>
>
> On Thu, Aug 3, 2017 at 6:24 AM, Ufuk Celebi <[hidden email]> wrote:
>
>> I haven't followed this discussion in detail nor am I familiar with
>> the service authorization topic or Flakka, but a) sounds like a lot of
>> maintenance work to me.
>>
>> If possible I would go with c) and maybe start a discussion about
>> dropping Scala 2.10 support to check whether that is a viable option
>> or not.
>>
>> – Ufuk
>>
>>
>> On Thu, Aug 3, 2017 at 1:59 PM, Till Rohrmann <[hidden email]>
>> wrote:
>> > Alternatively there would also be an option
>> >
>> > c) only support mutual auth for Akka 2.4+ if the backport is
>> unrealistic to
>> > do
>> >
>> > But this of course would break security for Scala 2.10. On the other
>> hand
>> > people are already using Flink without this feature.
>> >
>> > Cheers,
>> > Till
>> >
>> > On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright <[hidden email]>
>> wrote:
>> >
>> >> Thanks Till and Aljoscha for the feedback.
>> >>
>> >> Seems there are two ways to proceed here, if we accept mutual SSL as
>> the
>> >> basis.
>> >>
>> >> a) Backport mutual-auth support from Akka 2.4 to Flakka.
>> >> b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4
>> (FLINK-3662).
>> >>
>> >> Let's assume (a) for now.
>> >>
>> >>
>> >>
>> >> On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <[hidden email]>
>> >> wrote:
>> >>
>> >> > Dropping Java 7 alone is not enough to move to Akka 2.4+. For that we
>> >> need
>> >> > at least Scala 2.11.
>> >> >
>> >> > Cheers,
>> >> > Till
>> >> >
>> >> > On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <
>> [hidden email]>
>> >> > wrote:
>> >> >
>> >> > > Hi Eron,
>> >> > >
>> >> > > I think after Dropping support for Java 7 we will move to Akka
>> 2.4+, so
>> >> > we
>> >> > > should be good there. I think quite some users should find a (more)
>> >> > secure
>> >> > > Flink interesting.
>> >> > >
>> >> > > Best,
>> >> > > Aljoscha
>> >> > > > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]>
>> wrote:
>> >> > > >
>> >> > > > Hello, now might be a good time to revisit an important
>> enhancement
>> >> to
>> >> > > > Flink security, so-called service authorization.   This means the
>> >> > > hardening
>> >> > > > of a Flink cluster against unauthorized use with some sort of
>> >> > > > authentication and authorization scheme.   Today, Flink relies
>> >> entirely
>> >> > > on
>> >> > > > network isolation to protect itself from unauthorized job
>> submission
>> >> > and
>> >> > > > control, and to protect the secrets contained within a Flink
>> cluster.
>> >> > > > This is a problem in multi-user environments like YARN/Mesos/K8.
>> >> > > >
>> >> > > > Last fall, an effort was made to implement service authorization
>> but
>> >> > the
>> >> > > PR
>> >> > > > was ultimately rejected.   The idea was to add a simple secret
>> key to
>> >> > all
>> >> > > > network communication between the client, JM, and TM.   Akka
>> itself
>> >> has
>> >> > > > such a feature which formed the basis of the solution.  There are
>> >> > > usability
>> >> > > > challenges with this solution, including a dependency on SSL.
>> >> > > >
>> >> > > > Since then, the situation has evolved somewhat, and the use of
>> SSL
>> >> > mutual
>> >> > > > authentication is more viable.   Mutual auth is supported in Akka
>> >> > 2.4.12+
>> >> > > > (or could be backported to Flakka).  My proposal is:
>> >> > > >
>> >> > > > 1. Upgrade Akka or backport the functionality to Flakka (see
>> commit
>> >> > > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
>> >> > > > 2. Implement SSL on any endpoint that doesn't yet support it
>> (e.g.
>> >> > > > queryable state).
>> >> > > > 3. Enable mutual auth in Akka and implement it on non-Akka
>> endpoints.
>> >> > > > 4. Implement a simple authorization layer that accepts any
>> >> > authenticated
>> >> > > > connection.
>> >> > > > 5. (stretch) generate and store a certificate automatically in
>> YARN
>> >> > mode.
>> >> > > > 6. (stretch) Develop an alternate authentication method for the
>> Web
>> >> UI.
>> >> > > >
>> >> > > > Are folks interested in this capability?  Thoughts on the use of
>> SSL
>> >> > > mutual
>> >> > > > auth versus something else?  Thanks!
>> >> > > >
>> >> > > > -Eron
>> >> > >
>> >> > >
>> >> >
>> >>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Service Authorization (redux)

Stephan Ewen
Hi!

Reviving this thread, thank you, Eron, for starting this and for the
preparation of the FLIP.
I am sharing some thoughts below, and some input based on what has changed
with FLIP-6 and the evolution of queryable state.

Best,
Stephan

-------------------------------

*Internal vs. External Connectivity*

That is a very helpful distinction, let's build on that.

  - I would suggest to treat eventually all communication coming
potentially from users as external, meaning Client-to-Dispatcher,
Client-to-JobManager (trigger savepoint, change parallelism, ...), Web UI,
Queryable State.

  - That leaves communication that is only between JobManager/TaskManager/
ResourceManager/Dispatcher/HistoryServer as internal.

  - I am somewhat operating under the assumption that all external
communication will eventually be HTTP/REST. That works best with many
setups and is the basis for using service proxies that handle
authentication/authorization.


In Flink 1.5 and future versions, we have the following update there:

  - Akka is now strictly internal connectivity, the client (except legacy
client) do not use it any more.

  - The Blob Server will move to purely internal connectivity in Flink 1.6,
where a POST of a job to the Dispatcher has the jars and the JobGraph. That
is important for Kubernetes setups, where exposing the BlobServer and
querying the blob port causes quite some friction.

  - Treating queryable state as "internal connectivity" is fine for now. We
should treat it as "external" connectivity in the future if we move it to
HTTP/REST.


*Internal Connectivity and SSL Mutual Authentication*

Simply activating SSL mutual authentication for the internal communication
is a really low hanging fruit.

Activating client authentication for Akka, network stack Netty (and Blob
Server/Client in Flink 1.6) should require no change in the configurations
with respect to Flink 1.4. All processes are, with respect to internal
communication, simultaneously server and client endpoints. Because of that,
they already need KeyStore and TrustStore files for SSL handshakes, where
the TrustStore needs to trust the KeyStore Certificate.

I personally favor the suggestion made to have a script that generates a
self-signed certificate and adds it to "conf" and updates the
configuration. That should be picked up by the Yarn and Mesos clients
anyways.


*External Connectivity*

There is a huge surface area and I think we need to give users a way to
plug in their own tools.
From what I see (and after some discussions with Patrick and Gary) I think
it makes sense to look at proxies in a broad way, similar to the approach
Eron outlined.

The basic approach could be like that:

  - Everything goes through HTTPS, so the proxy can work with HTTP headers.
  - The proxy handles authentication and possibly authorization. The proxy
adds some header, for example a user name, a group id, an authorization
token.
  - Flink can configure an implementation of an 'authorizer' or validator
on the headers to decide whether the request is valid.

  - Example 1: The proxy does authentication and adds the user name / group
as a header. The the Flink-side authorizer simply checks whether the name
is in the config (simple ACL-style) scheme.
  - Example 2: The proxy adds an JSON Web Token and the authorizer
validates that token.

For secure connections between the Proxy and the Flink Endpoint I would
follow Eron's suggestion, to use separate KeyStores and TrustStores than
for internal communication.

For Yarn and Mesos, I would like to see if we could handle those again as a
special case of the proxies above:
  - DCOS Admin Router forwards the user authentication token, so that could
be another authorizer implementation.
  - In YARN we could see if can implement the IP filter via such an
authorizer.


*Hostname Verification*

I am not sure if the suggestion in the FLIP means to not use hostname
verification at all, or to always use it (no configuration flag). Eron, can
you clarify what you mean there?

For Kubernetes, it is very hard to work with certificates and have hostname
verification on.

If we assume internal communication works strictly with a shared secret
certificate and with client authentication, does hostname verification
actually still add security in that particular setup? My understanding was
that  hostname verification is important to not have some valid certificate
presented, but the one bound to the server you want to talk to. If we have
anyways one trusted certificate only, isn't that already implied?

On the other hand, it is still possible (and potentially valuable) for
users in standalone mode to use keystores and truststores from a PKI, in
which case there may still be an argument in favor of hostname verification.



On Wed, Sep 27, 2017 at 6:46 AM, Eron Wright <[hidden email]> wrote:

> Hi folks, I'm happy to share with you a draft of a FLIP for service
> authorization.   As I mentioned at the top of this thread, the goal is to
> protect a deployed Flink cluster/session from unauthorized use.   In the
> doc, I propose the use of SSL client authentication for internal
> communication, plus YARN/Kubernetes/Mesos-specific functionality to achieve
> single sign-on for the web interface/API.
>
> FLIP-? - Service Authorization
> <https://docs.google.com/document/d/13IRPb2GdL842rIzMgEn0ibO
> QHNku6W8aMf1p7gCPJjg/edit?usp=sharing>
>
> Feel free to make comments and to give feedback on whether this feature
> would be useful to you.
>
> Thanks!
> Eron
>
>
> On Thu, Aug 3, 2017 at 11:11 AM, Eron Wright <[hidden email]> wrote:
>
> > Till, with (c) are you suggesting that we'd use Akka 2.3 for Scala 2.10
> > and Akka 2.4+ for Scala 2.11+?   Sounds reasonable but I don't know how
> > feasible it is.   I will say I'm optimistic because a) Akka 2.4 is said
> to
> > be binary compatible, and b) the Flakka fork appears to be subsumed by
> 2.4.
> >
> > Let us then take (c) as the tentative plan.
> >
> > I agree the community should discuss dropping Scala 2.10 but I don't want
> > to drive that conversation.
> >
> > Thanks
> >
> >
> > On Thu, Aug 3, 2017 at 6:24 AM, Ufuk Celebi <[hidden email]> wrote:
> >
> >> I haven't followed this discussion in detail nor am I familiar with
> >> the service authorization topic or Flakka, but a) sounds like a lot of
> >> maintenance work to me.
> >>
> >> If possible I would go with c) and maybe start a discussion about
> >> dropping Scala 2.10 support to check whether that is a viable option
> >> or not.
> >>
> >> – Ufuk
> >>
> >>
> >> On Thu, Aug 3, 2017 at 1:59 PM, Till Rohrmann <[hidden email]>
> >> wrote:
> >> > Alternatively there would also be an option
> >> >
> >> > c) only support mutual auth for Akka 2.4+ if the backport is
> >> unrealistic to
> >> > do
> >> >
> >> > But this of course would break security for Scala 2.10. On the other
> >> hand
> >> > people are already using Flink without this feature.
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Wed, Aug 2, 2017 at 7:21 PM, Eron Wright <[hidden email]>
> >> wrote:
> >> >
> >> >> Thanks Till and Aljoscha for the feedback.
> >> >>
> >> >> Seems there are two ways to proceed here, if we accept mutual SSL as
> >> the
> >> >> basis.
> >> >>
> >> >> a) Backport mutual-auth support from Akka 2.4 to Flakka.
> >> >> b) Drop support for Scala 2.10 (FLINK-?), move to Akka 2.4
> >> (FLINK-3662).
> >> >>
> >> >> Let's assume (a) for now.
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Aug 1, 2017 at 3:05 PM, Till Rohrmann <[hidden email]>
> >> >> wrote:
> >> >>
> >> >> > Dropping Java 7 alone is not enough to move to Akka 2.4+. For that
> we
> >> >> need
> >> >> > at least Scala 2.11.
> >> >> >
> >> >> > Cheers,
> >> >> > Till
> >> >> >
> >> >> > On Tue, Aug 1, 2017 at 4:22 PM, Aljoscha Krettek <
> >> [hidden email]>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi Eron,
> >> >> > >
> >> >> > > I think after Dropping support for Java 7 we will move to Akka
> >> 2.4+, so
> >> >> > we
> >> >> > > should be good there. I think quite some users should find a
> (more)
> >> >> > secure
> >> >> > > Flink interesting.
> >> >> > >
> >> >> > > Best,
> >> >> > > Aljoscha
> >> >> > > > On 24. Jul 2017, at 03:11, Eron Wright <[hidden email]>
> >> wrote:
> >> >> > > >
> >> >> > > > Hello, now might be a good time to revisit an important
> >> enhancement
> >> >> to
> >> >> > > > Flink security, so-called service authorization.   This means
> the
> >> >> > > hardening
> >> >> > > > of a Flink cluster against unauthorized use with some sort of
> >> >> > > > authentication and authorization scheme.   Today, Flink relies
> >> >> entirely
> >> >> > > on
> >> >> > > > network isolation to protect itself from unauthorized job
> >> submission
> >> >> > and
> >> >> > > > control, and to protect the secrets contained within a Flink
> >> cluster.
> >> >> > > > This is a problem in multi-user environments like
> YARN/Mesos/K8.
> >> >> > > >
> >> >> > > > Last fall, an effort was made to implement service
> authorization
> >> but
> >> >> > the
> >> >> > > PR
> >> >> > > > was ultimately rejected.   The idea was to add a simple secret
> >> key to
> >> >> > all
> >> >> > > > network communication between the client, JM, and TM.   Akka
> >> itself
> >> >> has
> >> >> > > > such a feature which formed the basis of the solution.  There
> are
> >> >> > > usability
> >> >> > > > challenges with this solution, including a dependency on SSL.
> >> >> > > >
> >> >> > > > Since then, the situation has evolved somewhat, and the use of
> >> SSL
> >> >> > mutual
> >> >> > > > authentication is more viable.   Mutual auth is supported in
> Akka
> >> >> > 2.4.12+
> >> >> > > > (or could be backported to Flakka).  My proposal is:
> >> >> > > >
> >> >> > > > 1. Upgrade Akka or backport the functionality to Flakka (see
> >> commit
> >> >> > > > 5d03902c5ec3212cd28f26c9b3ef7c3b628b9451).
> >> >> > > > 2. Implement SSL on any endpoint that doesn't yet support it
> >> (e.g.
> >> >> > > > queryable state).
> >> >> > > > 3. Enable mutual auth in Akka and implement it on non-Akka
> >> endpoints.
> >> >> > > > 4. Implement a simple authorization layer that accepts any
> >> >> > authenticated
> >> >> > > > connection.
> >> >> > > > 5. (stretch) generate and store a certificate automatically in
> >> YARN
> >> >> > mode.
> >> >> > > > 6. (stretch) Develop an alternate authentication method for the
> >> Web
> >> >> UI.
> >> >> > > >
> >> >> > > > Are folks interested in this capability?  Thoughts on the use
> of
> >> SSL
> >> >> > > mutual
> >> >> > > > auth versus something else?  Thanks!
> >> >> > > >
> >> >> > > > -Eron
> >> >> > >
> >> >> > >
> >> >> >
> >> >>
> >>
> >
> >
>