Contributing to Flink

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Contributing to Flink

Rai, Niraj
Hi Flink Dev,
I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink.
Thanks
Niraj

Reply | Threaded
Open this post in threaded view
|

Re: Contributing to Flink

Henry Saputra
Hi Niraj,

Thanks for your interest at Apache Flink. The quickest is to just give
Flink a spin and figure out how it works.
This would get you start on how it works before actually doing work on Flink =)

Please do visit Flink how to contribute page [1] and subscribe to dev
mailing list [2] to start following up.

Welcome =)

[1] http://flink.apache.org/how-to-contribute.html
[2] http://flink.apache.org/community.html#mailing-lists

On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj <[hidden email]> wrote:
> Hi Flink Dev,
> I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink.
> Thanks
> Niraj
>
Reply | Threaded
Open this post in threaded view
|

Re: Contributing to Flink

Robert Metzger
Hi Niraj,

Welcome to the Flink community ;)
I'm really excited that you want to contribute to our project, and since
you've asked for something in the security area, I actually have something
very concrete in mind.
We recently added some support for accessing (Kerberos) secured HDFS
clusters in Flink: https://issues.apache.org/jira/browse/FLINK-1504.
However, the implementation is very simple because it assumes that every
worker of Flink (TaskManager) is authenticated with Kerberos (kinit). Its
not very practical for large setups because you have to ssh to all machines
to log into Kerberos.

What I would really like to have in Flink would be an way to transfer the
authentication tokens form the JobManager (master) to the TaskManagers.
This way, users only have to be authenticated with Kerberos at the
JobManager, and Flink is taking care of the rest.
As far as I understood it, Hadoop has already all the utilities in place
for getting and transferring the delegation tokens.
Max Michels, another committer in our project has quite a good
understanding of the details there. It would be great if you (Max) could
chime in if I forgot something.

If you are interested in working on this, you can file a JIRA (
https://issues.apache.org/jira/browse/FLINK) for tracking the progress and
discussing the details.
If not I'm sure we'll come up with more interesting ideas.


Robert







On Thu, Feb 26, 2015 at 11:07 PM, Henry Saputra <[hidden email]>
wrote:

> Hi Niraj,
>
> Thanks for your interest at Apache Flink. The quickest is to just give
> Flink a spin and figure out how it works.
> This would get you start on how it works before actually doing work on
> Flink =)
>
> Please do visit Flink how to contribute page [1] and subscribe to dev
> mailing list [2] to start following up.
>
> Welcome =)
>
> [1] http://flink.apache.org/how-to-contribute.html
> [2] http://flink.apache.org/community.html#mailing-lists
>
> On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj <[hidden email]> wrote:
> > Hi Flink Dev,
> > I am looking to contribute to Flink, especially in the area of security.
> In the past, I have contributed to Pig, Hive and HDFS. I would really
> appreciate, if I can get some work assigned to me. Looking forward to hear
> back from the development community of Flink.
> > Thanks
> > Niraj
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: Contributing to Flink

Rai, Niraj
Hi Robert,
Thanks for the detailed response. I worked on the encryption of HDFS as well as the crypto file system in HDFS, so, I am aware of how it is done in Hadoop. Let me sync up with Max to get started on it.
I will also start looking into the current implementations.
Niraj


From: Robert Metzger [mailto:[hidden email]]
Sent: Thursday, February 26, 2015 3:11 PM
To: [hidden email]
Cc: Rai, Niraj
Subject: Re: Contributing to Flink

Hi Niraj,

Welcome to the Flink community ;)
I'm really excited that you want to contribute to our project, and since you've asked for something in the security area, I actually have something very concrete in mind.
We recently added some support for accessing (Kerberos) secured HDFS clusters in Flink: https://issues.apache.org/jira/browse/FLINK-1504.
However, the implementation is very simple because it assumes that every worker of Flink (TaskManager) is authenticated with Kerberos (kinit). Its not very practical for large setups because you have to ssh to all machines to log into Kerberos.

What I would really like to have in Flink would be an way to transfer the authentication tokens form the JobManager (master) to the TaskManagers. This way, users only have to be authenticated with Kerberos at the JobManager, and Flink is taking care of the rest.
As far as I understood it, Hadoop has already all the utilities in place for getting and transferring the delegation tokens.
Max Michels, another committer in our project has quite a good understanding of the details there. It would be great if you (Max) could chime in if I forgot something.

If you are interested in working on this, you can file a JIRA (https://issues.apache.org/jira/browse/FLINK) for tracking the progress and discussing the details.
If not I'm sure we'll come up with more interesting ideas.


Robert







On Thu, Feb 26, 2015 at 11:07 PM, Henry Saputra <[hidden email]<mailto:[hidden email]>> wrote:
Hi Niraj,

Thanks for your interest at Apache Flink. The quickest is to just give
Flink a spin and figure out how it works.
This would get you start on how it works before actually doing work on Flink =)

Please do visit Flink how to contribute page [1] and subscribe to dev
mailing list [2] to start following up.

Welcome =)

[1] http://flink.apache.org/how-to-contribute.html
[2] http://flink.apache.org/community.html#mailing-lists

On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj <[hidden email]<mailto:[hidden email]>> wrote:
> Hi Flink Dev,
> I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink.
> Thanks
> Niraj
>

mxm
Reply | Threaded
Open this post in threaded view
|

Re: Contributing to Flink

mxm
Hi Niraj,

Pleased to here you want to start contributing to Flink :)

In terms of security, there are some open issues. Like Robert
metioned, it would be great if you could implement proper HDFS
Kerberos authentication. Basically, the HDFS Delegation Token needs to
be transferred to the workers so that they don't have to be
authenticated themselves. Also, proper renewal of the Token needs to
be taken care of.

Another security task could be to implement authentication support in
Flink. This could be done by asking the user for some kind of shared
secret. Alternatively, authentication could also be performed by
Kerberos' authentication method.

Let me know what you find interesting.

Best regards,
Max

On Fri, Feb 27, 2015 at 2:31 AM, Rai, Niraj <[hidden email]> wrote:

> Hi Robert,
> Thanks for the detailed response. I worked on the encryption of HDFS as well as the crypto file system in HDFS, so, I am aware of how it is done in Hadoop. Let me sync up with Max to get started on it.
> I will also start looking into the current implementations.
> Niraj
>
>
> From: Robert Metzger [mailto:[hidden email]]
> Sent: Thursday, February 26, 2015 3:11 PM
> To: [hidden email]
> Cc: Rai, Niraj
> Subject: Re: Contributing to Flink
>
> Hi Niraj,
>
> Welcome to the Flink community ;)
> I'm really excited that you want to contribute to our project, and since you've asked for something in the security area, I actually have something very concrete in mind.
> We recently added some support for accessing (Kerberos) secured HDFS clusters in Flink: https://issues.apache.org/jira/browse/FLINK-1504.
> However, the implementation is very simple because it assumes that every worker of Flink (TaskManager) is authenticated with Kerberos (kinit). Its not very practical for large setups because you have to ssh to all machines to log into Kerberos.
>
> What I would really like to have in Flink would be an way to transfer the authentication tokens form the JobManager (master) to the TaskManagers. This way, users only have to be authenticated with Kerberos at the JobManager, and Flink is taking care of the rest.
> As far as I understood it, Hadoop has already all the utilities in place for getting and transferring the delegation tokens.
> Max Michels, another committer in our project has quite a good understanding of the details there. It would be great if you (Max) could chime in if I forgot something.
>
> If you are interested in working on this, you can file a JIRA (https://issues.apache.org/jira/browse/FLINK) for tracking the progress and discussing the details.
> If not I'm sure we'll come up with more interesting ideas.
>
>
> Robert
>
>
>
>
>
>
>
> On Thu, Feb 26, 2015 at 11:07 PM, Henry Saputra <[hidden email]<mailto:[hidden email]>> wrote:
> Hi Niraj,
>
> Thanks for your interest at Apache Flink. The quickest is to just give
> Flink a spin and figure out how it works.
> This would get you start on how it works before actually doing work on Flink =)
>
> Please do visit Flink how to contribute page [1] and subscribe to dev
> mailing list [2] to start following up.
>
> Welcome =)
>
> [1] http://flink.apache.org/how-to-contribute.html
> [2] http://flink.apache.org/community.html#mailing-lists
>
> On Thu, Feb 26, 2015 at 1:45 PM, Rai, Niraj <[hidden email]<mailto:[hidden email]>> wrote:
>> Hi Flink Dev,
>> I am looking to contribute to Flink, especially in the area of security. In the past, I have contributed to Pig, Hive and HDFS. I would really appreciate, if I can get some work assigned to me. Looking forward to hear back from the development community of Flink.
>> Thanks
>> Niraj
>>
>