[HEADS-UP] Dev Week 33

[Aljoscha Krettek-2]

Hi Folks,
I just had an idea that I want to try now. The idea is to give little
HEADS-UP emails roughly at the beginning of each week. I would give a very
short summary of the big points that I want to work on for that week and if
others want to jump in they can also quickly outline what they are doing.

The reason I'm proposing this is that it can be hard to follow what is
going on in Flink. One could try and follow all the Jira issues and see who
is assigned to what but that would only give a rough indication. Currently,
people are working away on a new feature, sometimes for several weeks,
until they make a big code drop in a PR. I though it would be nice to know
roughly who's doing what. This would also give people the opportunity to
get involved in something if they find it interesting. If people find this
interesting I'll try and do this every week from now on.

Here goes my first heads-up: This (and the last weeks) week Stefan and I
have been working on the issues around
https://issues.apache.org/jira/browse/FLINK-3755. We were mostly
resurrecting an old PR from Till. Now we got it into good shape, we just
opened a big PR (https://github.com/apache/flink/pull/2376) that introduces
API for key-groups and support in some layers. For the rest of the week,
we're planning to refactor the state backends to make them work with
key-group aware state.

Happy coding :-)
Aljoscha

[Stephan Ewen]

Nice idea, I like that. I'll join with some HEADS-UP:

  - Till and me started working on FLIP-6 (rework the Yarn/Mesos/standalone
mode) together with some contributors from Alibaba. The work is currently
in a feature branch.

  - As part of that rework, we experimented with a new RPC abstraction that
builds on top of Akka. Think of it as "Flink-flavored typed actors". It has
actually turned out to be very helpful, we plan to write about it in a bit
(once it has converged)

  - Other than that, Ufuk and me did some work on unifying checkpoints and
savepoints. That will make it easier to recover on another cluster from a
regular checkpoint (in case the system gets stuck) and that a regular
savepoint will be used for recovery in the program that created it.





On Tue, Aug 16, 2016 at 10:34 PM, Aljoscha Krettek <[hidden email]>
wrote:

> Hi Folks,
> I just had an idea that I want to try now. The idea is to give little
> HEADS-UP emails roughly at the beginning of each week. I would give a very
> short summary of the big points that I want to work on for that week and if
> others want to jump in they can also quickly outline what they are doing.
>
> The reason I'm proposing this is that it can be hard to follow what is
> going on in Flink. One could try and follow all the Jira issues and see who
> is assigned to what but that would only give a rough indication. Currently,
> people are working away on a new feature, sometimes for several weeks,
> until they make a big code drop in a PR. I though it would be nice to know
> roughly who's doing what. This would also give people the opportunity to
> get involved in something if they find it interesting. If people find this
> interesting I'll try and do this every week from now on.
>
> Here goes my first heads-up: This (and the last weeks) week Stefan and I
> have been working on the issues around
> https://issues.apache.org/jira/browse/FLINK-3755. We were mostly
> resurrecting an old PR from Till. Now we got it into good shape, we just
> opened a big PR (https://github.com/apache/flink/pull/2376) that
> introduces
> API for key-groups and support in some layers. For the rest of the week,
> we're planning to refactor the state backends to make them work with
> key-group aware state.
>
> Happy coding :-)
> Aljoscha
>

[Till Rohrmann]

Good initiative @Aljoscha.

- Just a convenience link for FLIP-6: The feature branch can be found here
https://github.com/apache/flink/tree/flip-6.

- My main task is the review of the Mesos PR of Eron (
https://github.com/apache/flink/pull/2315) so that we can continue making
progress there.

On Wed, Aug 17, 2016 at 10:14 AM, Stephan Ewen <[hidden email]> wrote:

> Nice idea, I like that. I'll join with some HEADS-UP:
>
>   - Till and me started working on FLIP-6 (rework the Yarn/Mesos/standalone
> mode) together with some contributors from Alibaba. The work is currently
> in a feature branch.
>
>   - As part of that rework, we experimented with a new RPC abstraction that
> builds on top of Akka. Think of it as "Flink-flavored typed actors". It has
> actually turned out to be very helpful, we plan to write about it in a bit
> (once it has converged)
>
>   - Other than that, Ufuk and me did some work on unifying checkpoints and
> savepoints. That will make it easier to recover on another cluster from a
> regular checkpoint (in case the system gets stuck) and that a regular
> savepoint will be used for recovery in the program that created it.
>
>
>
>
>
> On Tue, Aug 16, 2016 at 10:34 PM, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > Hi Folks,
> > I just had an idea that I want to try now. The idea is to give little
> > HEADS-UP emails roughly at the beginning of each week. I would give a
> very
> > short summary of the big points that I want to work on for that week and
> if
> > others want to jump in they can also quickly outline what they are doing.
> >
> > The reason I'm proposing this is that it can be hard to follow what is
> > going on in Flink. One could try and follow all the Jira issues and see
> who
> > is assigned to what but that would only give a rough indication.
> Currently,
> > people are working away on a new feature, sometimes for several weeks,
> > until they make a big code drop in a PR. I though it would be nice to
> know
> > roughly who's doing what. This would also give people the opportunity to
> > get involved in something if they find it interesting. If people find
> this
> > interesting I'll try and do this every week from now on.
> >
> > Here goes my first heads-up: This (and the last weeks) week Stefan and I
> > have been working on the issues around
> > https://issues.apache.org/jira/browse/FLINK-3755. We were mostly
> > resurrecting an old PR from Till. Now we got it into good shape, we just
> > opened a big PR (https://github.com/apache/flink/pull/2376) that
> > introduces
> > API for key-groups and support in some layers. For the rest of the week,
> > we're planning to refactor the state backends to make them work with
> > key-group aware state.
> >
> > Happy coding :-)
> > Aljoscha
> >
>

[Ufuk Celebi-2]

I've seen this in BEAM as well and I really like it.

- I'm finishing up the initial work for FLIP-3 (Docs restructuring)
and hope to post something to the mailing list later today about it.

- I will start working on top of Aljoscha's and Stefan's pull request
#2376 with an initial version for non-partitioned rescalable state. I
think that this will continue into next week.

- Other than that, I want to write down a shirt FLIP for exposing the
savepoint/checkpoint unification to the user (internal refactoring is
an open PR #2366).


On Wed, Aug 17, 2016 at 10:39 AM, Till Rohrmann <[hidden email]> wrote:

> Good initiative @Aljoscha.
>
> - Just a convenience link for FLIP-6: The feature branch can be found here
> https://github.com/apache/flink/tree/flip-6.
>
> - My main task is the review of the Mesos PR of Eron (
> https://github.com/apache/flink/pull/2315) so that we can continue making
> progress there.
>
> On Wed, Aug 17, 2016 at 10:14 AM, Stephan Ewen <[hidden email]> wrote:
>
>> Nice idea, I like that. I'll join with some HEADS-UP:
>>
>>   - Till and me started working on FLIP-6 (rework the Yarn/Mesos/standalone
>> mode) together with some contributors from Alibaba. The work is currently
>> in a feature branch.
>>
>>   - As part of that rework, we experimented with a new RPC abstraction that
>> builds on top of Akka. Think of it as "Flink-flavored typed actors". It has
>> actually turned out to be very helpful, we plan to write about it in a bit
>> (once it has converged)
>>
>>   - Other than that, Ufuk and me did some work on unifying checkpoints and
>> savepoints. That will make it easier to recover on another cluster from a
>> regular checkpoint (in case the system gets stuck) and that a regular
>> savepoint will be used for recovery in the program that created it.
>>
>>
>>
>>
>>
>> On Tue, Aug 16, 2016 at 10:34 PM, Aljoscha Krettek <[hidden email]>
>> wrote:
>>
>> > Hi Folks,
>> > I just had an idea that I want to try now. The idea is to give little
>> > HEADS-UP emails roughly at the beginning of each week. I would give a
>> very
>> > short summary of the big points that I want to work on for that week and
>> if
>> > others want to jump in they can also quickly outline what they are doing.
>> >
>> > The reason I'm proposing this is that it can be hard to follow what is
>> > going on in Flink. One could try and follow all the Jira issues and see
>> who
>> > is assigned to what but that would only give a rough indication.
>> Currently,
>> > people are working away on a new feature, sometimes for several weeks,
>> > until they make a big code drop in a PR. I though it would be nice to
>> know
>> > roughly who's doing what. This would also give people the opportunity to
>> > get involved in something if they find it interesting. If people find
>> this
>> > interesting I'll try and do this every week from now on.
>> >
>> > Here goes my first heads-up: This (and the last weeks) week Stefan and I
>> > have been working on the issues around
>> > https://issues.apache.org/jira/browse/FLINK-3755. We were mostly
>> > resurrecting an old PR from Till. Now we got it into good shape, we just
>> > opened a big PR (https://github.com/apache/flink/pull/2376) that
>> > introduces
>> > API for key-groups and support in some layers. For the rest of the week,
>> > we're planning to refactor the state backends to make them work with
>> > key-group aware state.
>> >
>> > Happy coding :-)
>> > Aljoscha
>> >
>>