[DISCUSS] Secure Flink clusters

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Secure Flink clusters

Eron Wright
Hello!

There’s been a few discussions lately on how to improve the Kerberos support in Flink.  I’ve drafted a design document that lays out a plan to support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In addition, the plan contemplates secure, TLS-based communication between cluster components.

The main goals are secure data access for Kerberized connectors and cluster authentication to prevent unauthorized access to cluster secrets.

Here is the document:
https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing

I anticipate filing a multitude of JIRAs following a design discussion.    It is a big task and there will be opportunities for others in the community to help.

Thanks,
Eron Wright
EMC
mxm
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Secure Flink clusters

mxm
Hi Eron,

Thank you for this comprehensive design document. Really great read.
I've left some minor comments.

+1 for breaking down the tasks into many JIRA issues; we have quite
some ambitious plans now :) It would be great to get some more people
from the community involved as well.

Best,
Max

On Wed, May 11, 2016 at 9:09 AM, Wright, Eron <[hidden email]> wrote:

> Hello!
>
> There’s been a few discussions lately on how to improve the Kerberos support in Flink.  I’ve drafted a design document that lays out a plan to support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In addition, the plan contemplates secure, TLS-based communication between cluster components.
>
> The main goals are secure data access for Kerberized connectors and cluster authentication to prevent unauthorized access to cluster secrets.
>
> Here is the document:
> https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
>
> I anticipate filing a multitude of JIRAs following a design discussion.    It is a big task and there will be opportunities for others in the community to help.
>
> Thanks,
> Eron Wright
> EMC
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Secure Flink clusters

Robert Metzger
Hi Eron,

thanks a lot for putting so much effort into the design document. You've
probably spend a lot of time to come up with it!
I have to admit that I'm not that familiar with the topic, so I probably
need to re-read it again to digest it completely.

What are your plans for implementing the proposed changes? (time-wise and
people-wise?) I'm asking to get an idea of when we can expect the changes
in the master, in releases, ...

I think Stefano Baghino also had some discussions about improving Flink's
security on the mailing list recently. Maybe you guys can sync your efforts
and collaborate on this.

Regards,
Robert


On Fri, May 13, 2016 at 12:47 PM, Maximilian Michels <[hidden email]> wrote:

> Hi Eron,
>
> Thank you for this comprehensive design document. Really great read.
> I've left some minor comments.
>
> +1 for breaking down the tasks into many JIRA issues; we have quite
> some ambitious plans now :) It would be great to get some more people
> from the community involved as well.
>
> Best,
> Max
>
> On Wed, May 11, 2016 at 9:09 AM, Wright, Eron <[hidden email]> wrote:
> > Hello!
> >
> > There’s been a few discussions lately on how to improve the Kerberos
> support in Flink.  I’ve drafted a design document that lays out a plan to
> support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In
> addition, the plan contemplates secure, TLS-based communication between
> cluster components.
> >
> > The main goals are secure data access for Kerberized connectors and
> cluster authentication to prevent unauthorized access to cluster secrets.
> >
> > Here is the document:
> >
> https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
> >
> > I anticipate filing a multitude of JIRAs following a design discussion.
>   It is a big task and there will be opportunities for others in the
> community to help.
> >
> > Thanks,
> > Eron Wright
> > EMC
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Secure Flink clusters

Eron Wright
Thanks to all who reviewed the document.    It appears we have a good plan and I'm filing JIRA issues accordingly.

Robert, I'm in touch with Max, Stephan, and Stefano.    I’ll update the thread when we have a better sense of the timing.   The work will clearly span a couple of releases.

Eron


> On May 17, 2016, at 8:35 AM, Robert Metzger <[hidden email]> wrote:
>
> Hi Eron,
>
> thanks a lot for putting so much effort into the design document. You've
> probably spend a lot of time to come up with it!
> I have to admit that I'm not that familiar with the topic, so I probably
> need to re-read it again to digest it completely.
>
> What are your plans for implementing the proposed changes? (time-wise and
> people-wise?) I'm asking to get an idea of when we can expect the changes
> in the master, in releases, ...
>
> I think Stefano Baghino also had some discussions about improving Flink's
> security on the mailing list recently. Maybe you guys can sync your efforts
> and collaborate on this.
>
> Regards,
> Robert
>
>
> On Fri, May 13, 2016 at 12:47 PM, Maximilian Michels <[hidden email]> wrote:
>
>> Hi Eron,
>>
>> Thank you for this comprehensive design document. Really great read.
>> I've left some minor comments.
>>
>> +1 for breaking down the tasks into many JIRA issues; we have quite
>> some ambitious plans now :) It would be great to get some more people
>> from the community involved as well.
>>
>> Best,
>> Max
>>
>> On Wed, May 11, 2016 at 9:09 AM, Wright, Eron <[hidden email]> wrote:
>>> Hello!
>>>
>>> There’s been a few discussions lately on how to improve the Kerberos
>> support in Flink.  I’ve drafted a design document that lays out a plan to
>> support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In
>> addition, the plan contemplates secure, TLS-based communication between
>> cluster components.
>>>
>>> The main goals are secure data access for Kerberized connectors and
>> cluster authentication to prevent unauthorized access to cluster secrets.
>>>
>>> Here is the document:
>>>
>> https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
>>>
>>> I anticipate filing a multitude of JIRAs following a design discussion.
>>  It is a big task and there will be opportunities for others in the
>> community to help.
>>>
>>> Thanks,
>>> Eron Wright
>>> EMC
>>

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Secure Flink clusters

Henry Saputra
Eron,

Could you please do also loop me in in the early discussions since we are
interested on deploying Flink as standalone to access secure data via
Kerberized access.

I also was talking to Owen from HDFS at the Apache Big Data and there could
be some work we can ask to be done in the Hadoop common or HDFS side.


- Henry

On Tue, May 17, 2016 at 11:10 AM, Wright, Eron <[hidden email]> wrote:

> Thanks to all who reviewed the document.    It appears we have a good plan
> and I'm filing JIRA issues accordingly.
>
> Robert, I'm in touch with Max, Stephan, and Stefano.    I’ll update the
> thread when we have a better sense of the timing.   The work will clearly
> span a couple of releases.
>
> Eron
>
>
> > On May 17, 2016, at 8:35 AM, Robert Metzger <[hidden email]> wrote:
> >
> > Hi Eron,
> >
> > thanks a lot for putting so much effort into the design document. You've
> > probably spend a lot of time to come up with it!
> > I have to admit that I'm not that familiar with the topic, so I probably
> > need to re-read it again to digest it completely.
> >
> > What are your plans for implementing the proposed changes? (time-wise and
> > people-wise?) I'm asking to get an idea of when we can expect the changes
> > in the master, in releases, ...
> >
> > I think Stefano Baghino also had some discussions about improving Flink's
> > security on the mailing list recently. Maybe you guys can sync your
> efforts
> > and collaborate on this.
> >
> > Regards,
> > Robert
> >
> >
> > On Fri, May 13, 2016 at 12:47 PM, Maximilian Michels <[hidden email]>
> wrote:
> >
> >> Hi Eron,
> >>
> >> Thank you for this comprehensive design document. Really great read.
> >> I've left some minor comments.
> >>
> >> +1 for breaking down the tasks into many JIRA issues; we have quite
> >> some ambitious plans now :) It would be great to get some more people
> >> from the community involved as well.
> >>
> >> Best,
> >> Max
> >>
> >> On Wed, May 11, 2016 at 9:09 AM, Wright, Eron <[hidden email]> wrote:
> >>> Hello!
> >>>
> >>> There’s been a few discussions lately on how to improve the Kerberos
> >> support in Flink.  I’ve drafted a design document that lays out a plan
> to
> >> support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In
> >> addition, the plan contemplates secure, TLS-based communication between
> >> cluster components.
> >>>
> >>> The main goals are secure data access for Kerberized connectors and
> >> cluster authentication to prevent unauthorized access to cluster
> secrets.
> >>>
> >>> Here is the document:
> >>>
> >>
> https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
> >>>
> >>> I anticipate filing a multitude of JIRAs following a design discussion.
> >>  It is a big task and there will be opportunities for others in the
> >> community to help.
> >>>
> >>> Thanks,
> >>> Eron Wright
> >>> EMC
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Secure Flink clusters

Eron Wright
In reply to this post by Eron Wright
Update, the following issues were filed:

- [FLINK-3929] Support for Kerberos Authentication with Keytab Credential
- [FLINK-3930] Implement Service-Level Authorization
- [FLINK-3931] Implement Transport Encryption (SSL/TLS)
- [FLINK-3932] Implement State Backend Security

> On May 17, 2016, at 11:10 AM, Wright, Eron <[hidden email]> wrote:
>
> Thanks to all who reviewed the document.    It appears we have a good plan and I'm filing JIRA issues accordingly.
>
> Robert, I'm in touch with Max, Stephan, and Stefano.    I’ll update the thread when we have a better sense of the timing.   The work will clearly span a couple of releases.
>
> Eron
>
>
>> On May 17, 2016, at 8:35 AM, Robert Metzger <[hidden email]> wrote:
>>
>> Hi Eron,
>>
>> thanks a lot for putting so much effort into the design document. You've
>> probably spend a lot of time to come up with it!
>> I have to admit that I'm not that familiar with the topic, so I probably
>> need to re-read it again to digest it completely.
>>
>> What are your plans for implementing the proposed changes? (time-wise and
>> people-wise?) I'm asking to get an idea of when we can expect the changes
>> in the master, in releases, ...
>>
>> I think Stefano Baghino also had some discussions about improving Flink's
>> security on the mailing list recently. Maybe you guys can sync your efforts
>> and collaborate on this.
>>
>> Regards,
>> Robert
>>
>>
>> On Fri, May 13, 2016 at 12:47 PM, Maximilian Michels <[hidden email]> wrote:
>>
>>> Hi Eron,
>>>
>>> Thank you for this comprehensive design document. Really great read.
>>> I've left some minor comments.
>>>
>>> +1 for breaking down the tasks into many JIRA issues; we have quite
>>> some ambitious plans now :) It would be great to get some more people
>>> from the community involved as well.
>>>
>>> Best,
>>> Max
>>>
>>> On Wed, May 11, 2016 at 9:09 AM, Wright, Eron <[hidden email]> wrote:
>>>> Hello!
>>>>
>>>> There’s been a few discussions lately on how to improve the Kerberos
>>> support in Flink.  I’ve drafted a design document that lays out a plan to
>>> support keytab-based authentication for HDFS, Kafka, and ZooKeeper.  In
>>> addition, the plan contemplates secure, TLS-based communication between
>>> cluster components.
>>>>
>>>> The main goals are secure data access for Kerberized connectors and
>>> cluster authentication to prevent unauthorized access to cluster secrets.
>>>>
>>>> Here is the document:
>>>>
>>> https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing
>>>>
>>>> I anticipate filing a multitude of JIRAs following a design discussion.
>>> It is a big task and there will be opportunities for others in the
>>> community to help.
>>>>
>>>> Thanks,
>>>> Eron Wright
>>>> EMC
>>>
>