(DEPRECATED) Apache Flink Mailing List archive.

Flink internals

Classic

List

Threaded

6 messages Options

John Tipper

Flink internals

Hi all,

Is there a guide somewhere for the internals of Flink for developers wanting to get involved in core development? I'm particularly interested in any notes on how the codebase is put together so that it's possible to learn how it works internally.

I'm particularly interested in what classes are involved in the barrier mechanism and state and where these are found, but it's a large codebase so I'm sure it would be helpful for new developers to have pointers to what does what inside the codebase or where to start looking for details of certain things.

Many thanks,

John

Fan Liya

Re: Flink internals

You can find some articles here:
https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals

Best,
Liya Fan

On Thu, May 30, 2019 at 6:29 PM John Tipper <[hidden email]> wrote:

> Hi all,
>
> Is there a guide somewhere for the internals of Flink for developers
> wanting to get involved in core development? I'm particularly interested
> in any notes on how the codebase is put together so that it's possible to
> learn how it works internally.
>
> I'm particularly interested in what classes are involved in the barrier
> mechanism and state and where these are found, but it's a large codebase so
> I'm sure it would be helpful for new developers to have pointers to what
> does what inside the codebase or where to start looking for details of
> certain things.
>
> Many thanks,
>
> John
>

Rong Rong

Re: Flink internals

Hi Fan, John,

The flink internal link[1] seems to be not updated in the last years.
I found out some higher level pages in the official documentation here[2]
however they are still very limited.

Is there any other well maintained, internal documentations for
contributors?
I think it might be a good idea to maintain a higher-level guidance doc for
new contributors about Flink internals?

Best,
Rong

----
[1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html

On Thu, May 30, 2019 at 4:48 AM Fan Liya <[hidden email]> wrote:

> You can find some articles here:
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>
> Best,
> Liya Fan
>
> On Thu, May 30, 2019 at 6:29 PM John Tipper <[hidden email]>
> wrote:
>
> > Hi all,
> >
> > Is there a guide somewhere for the internals of Flink for developers
> > wanting to get involved in core development? I'm particularly interested
> > in any notes on how the codebase is put together so that it's possible to
> > learn how it works internally.
> >
> > I'm particularly interested in what classes are involved in the barrier
> > mechanism and state and where these are found, but it's a large codebase
> so
> > I'm sure it would be helpful for new developers to have pointers to what
> > does what inside the codebase or where to start looking for details of
> > certain things.
> >
> > Many thanks,
> >
> > John
> >
>

Till Rohrmann

Re: Flink internals

Hi John,

unfortunately, there are no really good and up to date documents about
Flink's internals. There was some discussion about updating the internals
[1] but the community did decide against submitting it as a season of docs
project. I agree that we should update our documentation about Flink's
internals, though.

At the moment, the best way about learning about Flink is the code and the
user/dev ML where you can ask about concepts and how things work.

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Apache-Flink-at-Season-of-Docs-td28133.html

Cheers,
Till

On Fri, May 31, 2019 at 2:50 AM Rong Rong <[hidden email]> wrote:

> Hi Fan, John,
>
> The flink internal link[1] seems to be not updated in the last years.
> I found out some higher level pages in the official documentation here[2]
> however they are still very limited.
>
> Is there any other well maintained, internal documentations for
> contributors?
> I think it might be a good idea to maintain a higher-level guidance doc for
> new contributors about Flink internals?
>
> Best,
> Rong
>
> ----
> [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
> [2]
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html
>
> On Thu, May 30, 2019 at 4:48 AM Fan Liya <[hidden email]> wrote:
>
> > You can find some articles here:
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
> >
> > Best,
> > Liya Fan
> >
> > On Thu, May 30, 2019 at 6:29 PM John Tipper <[hidden email]>
> > wrote:
> >
> > > Hi all,
> > >
> > > Is there a guide somewhere for the internals of Flink for developers
> > > wanting to get involved in core development? I'm particularly
> interested
> > > in any notes on how the codebase is put together so that it's possible
> to
> > > learn how it works internally.
> > >
> > > I'm particularly interested in what classes are involved in the barrier
> > > mechanism and state and where these are found, but it's a large
> codebase
> > so
> > > I'm sure it would be helpful for new developers to have pointers to
> what
> > > does what inside the codebase or where to start looking for details of
> > > certain things.
> > >
> > > Many thanks,
> > >
> > > John
> > >
> >
>

John Tipper

Re: Flink internals

Hi Till, Fan & Rong,

Thanks for your feedback - I'd seen the Flink internal page but sadly, as Rong pointed out, it's pretty limited and not maintained. I'll ask on the mailing lists, but I think it would be really helpful if there were a guide for Flink developers who want to contribute to Flink, as opposed to the current documentation which is really aimed at Flink application developers (and is really pretty good). I'd happily contribute to this, but I can't find my way round the internals...

Many thanks,

John
________________________________
From: Till Rohrmann <[hidden email]>
Sent: 04 June 2019 09:12
To: dev
Subject: Re: Flink internals

Hi John,

unfortunately, there are no really good and up to date documents about
Flink's internals. There was some discussion about updating the internals
[1] but the community did decide against submitting it as a season of docs
project. I agree that we should update our documentation about Flink's
internals, though.

At the moment, the best way about learning about Flink is the code and the
user/dev ML where you can ask about concepts and how things work.

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Apache-Flink-at-Season-of-Docs-td28133.html

Cheers,
Till

On Fri, May 31, 2019 at 2:50 AM Rong Rong <[hidden email]> wrote:

Piotr Nowojski-3

Re: Flink internals

Hi,

You can also read the FLIP proposals. Unluckily, one that is very internal [1] about credit based flow control [2] was not published as an official FLIP :( Regarding network stack and some of the other topics, there are some blogs/Flink Forward talks as well.

Piotrek
[1] https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84/edit <https://docs.google.com/document/d/1chTOuOqe0sBsjldA_r-wXYeSIhU2zRGpUaTaik7QZ84/edit>
[2] https://issues.apache.org/jira/browse/FLINK-7282 <https://issues.apache.org/jira/browse/FLINK-7282>

> On 4 Jun 2019, at 11:55, John Tipper <[hidden email]> wrote:
>
> Hi Till, Fan & Rong,
>
> Thanks for your feedback - I'd seen the Flink internal page but sadly, as Rong pointed out, it's pretty limited and not maintained. I'll ask on the mailing lists, but I think it would be really helpful if there were a guide for Flink developers who want to contribute to Flink, as opposed to the current documentation which is really aimed at Flink application developers (and is really pretty good). I'd happily contribute to this, but I can't find my way round the internals...
>
> Many thanks,
>
> John
> ________________________________
> From: Till Rohrmann <[hidden email]>
> Sent: 04 June 2019 09:12
> To: dev
> Subject: Re: Flink internals
>
> Hi John,
>
> unfortunately, there are no really good and up to date documents about
> Flink's internals. There was some discussion about updating the internals
> [1] but the community did decide against submitting it as a season of docs
> project. I agree that we should update our documentation about Flink's
> internals, though.
>
> At the moment, the best way about learning about Flink is the code and the
> user/dev ML where you can ask about concepts and how things work.
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Apache-Flink-at-Season-of-Docs-td28133.html
>
> Cheers,
> Till
>
> On Fri, May 31, 2019 at 2:50 AM Rong Rong <[hidden email]> wrote:
>
>> Hi Fan, John,
>>
>> The flink internal link[1] seems to be not updated in the last years.
>> I found out some higher level pages in the official documentation here[2]
>> however they are still very limited.
>>
>> Is there any other well maintained, internal documentations for
>> contributors?
>> I think it might be a good idea to maintain a higher-level guidance doc for
>> new contributors about Flink internals?
>>
>> Best,
>> Rong
>>
>> ----
>> [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>> [2]
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.8/internals/components.html
>>
>> On Thu, May 30, 2019 at 4:48 AM Fan Liya <[hidden email]> wrote:
>>
>>> You can find some articles here:
>>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Internals
>>>
>>> Best,
>>> Liya Fan
>>>
>>> On Thu, May 30, 2019 at 6:29 PM John Tipper <[hidden email]>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Is there a guide somewhere for the internals of Flink for developers
>>>> wanting to get involved in core development? I'm particularly
>> interested
>>>> in any notes on how the codebase is put together so that it's possible
>> to
>>>> learn how it works internally.
>>>>
>>>> I'm particularly interested in what classes are involved in the barrier
>>>> mechanism and state and where these are found, but it's a large
>> codebase
>>> so
>>>> I'm sure it would be helpful for new developers to have pointers to
>> what
>>>> does what inside the codebase or where to start looking for details of
>>>> certain things.
>>>>
>>>> Many thanks,
>>>>
>>>> John
>>>>
>>>
>>