[DISCUSS] Releasing Flink 1.5.0

[Aljoscha Krettek-2]

Hi Everyone,

When we decided to do the 1.4.0 release a while back we did that to get a stable release out before putting in a couple of new features. Back then, some of those new features (FLIP-6, network stack changes, local state recovery) were almost ready and we wanted to do a shortened 1.5.0 development cycle to allow for those features to become ready and then do the next release.

We are now approaching the approximate time where we wanted to do the Flink 1.5.0 release so I would like to gauge where we are and gather opinions on how we should proceed now.

With this, I'd also like to propose myself as the release manager for 1.5.0 but I'm very happy to yield if someone else would be interested in doing that.

What do you think?

Best,
Aljoscha

[Tzu-Li (Gordon) Tai]

Hi Aljoscha,

Thanks for starting the discussion.

I think there’s a few connector related must-have improvements that we should get in before the feature freeze, since quite a few users have been asking for them:

[FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start offset
[FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
[FLINK-8516] Pluggable shard-to-subtask partitioning for FlinkKinesisConsumer
[FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer

These are still missing in the master branch. Only FLINK-5479 is still lacking a pull request.

Cheers,
Gordon

On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek ([hidden email]) wrote:
Hi Everyone,  

When we decided to do the 1.4.0 release a while back we did that to get a stable release out before putting in a couple of new features. Back then, some of those new features (FLIP-6, network stack changes, local state recovery) were almost ready and we wanted to do a shortened 1.5.0 development cycle to allow for those features to become ready and then do the next release.  

We are now approaching the approximate time where we wanted to do the Flink 1.5.0 release so I would like to gauge where we are and gather opinions on how we should proceed now.  

With this, I'd also like to propose myself as the release manager for 1.5.0 but I'm very happy to yield if someone else would be interested in doing that.  

What do you think?  

Best,  
Aljoscha

[Timo Walther-2]

Hi Aljoscha,

it would be great if we can include the first version of the SQL client
(see FLIP-24, Implementation Plan 1). I will open a PR this week. I
think we can merge this with explicit "experimental/alpha" status. It is
far away from feature completeness but will be a great tool for Flink
beginners.

In order to use the SQL client we would need to also add some table
sources with the new unified table factories (FLINK-8535), but this is
optional because a user can implement own table factories at the begining.

Regards,
Timo


Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:

> Hi Aljoscha,
>
> Thanks for starting the discussion.
>
> I think there’s a few connector related must-have improvements that we should get in before the feature freeze, since quite a few users have been asking for them:
>
> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up start offset
> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions
> [FLINK-8516] Pluggable shard-to-subtask partitioning for FlinkKinesisConsumer
> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>
> These are still missing in the master branch. Only FLINK-5479 is still lacking a pull request.
>
> Cheers,
> Gordon
>
> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek ([hidden email]) wrote:
> Hi Everyone,
>
> When we decided to do the 1.4.0 release a while back we did that to get a stable release out before putting in a couple of new features. Back then, some of those new features (FLIP-6, network stack changes, local state recovery) were almost ready and we wanted to do a shortened 1.5.0 development cycle to allow for those features to become ready and then do the next release.
>
> We are now approaching the approximate time where we wanted to do the Flink 1.5.0 release so I would like to gauge where we are and gather opinions on how we should proceed now.
>
> With this, I'd also like to propose myself as the release manager for 1.5.0 but I'm very happy to yield if someone else would be interested in doing that.
>
> What do you think?
>
> Best,
> Aljoscha

[Christophe Jolif]

Hi guys,

Sorry for jumping in, but I think

[FLINK-8101] Elasticsearch 6.X support
[FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
Elasticsearch 5.2+ client

 have long been awaited and there was one PR from me and from someone else
showing the interest ;) So if you could consider it for 1.5 that would be
great!

Thanks!
--
Christophe

On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]> wrote:

> Hi Aljoscha,
>
> it would be great if we can include the first version of the SQL client
> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
> we can merge this with explicit "experimental/alpha" status. It is far away
> from feature completeness but will be a great tool for Flink beginners.
>
> In order to use the SQL client we would need to also add some table
> sources with the new unified table factories (FLINK-8535), but this is
> optional because a user can implement own table factories at the begining.
>
> Regards,
> Timo
>
>
> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>
> Hi Aljoscha,
>>
>> Thanks for starting the discussion.
>>
>> I think there’s a few connector related must-have improvements that we
>> should get in before the feature freeze, since quite a few users have been
>> asking for them:
>>
>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set up
>> start offset
>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>> consider idle partitions
>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>> FlinkKinesisConsumer
>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>
>> These are still missing in the master branch. Only FLINK-5479 is still
>> lacking a pull request.
>>
>> Cheers,
>> Gordon
>>
>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek ([hidden email])
>> wrote:
>> Hi Everyone,
>>
>> When we decided to do the 1.4.0 release a while back we did that to get a
>> stable release out before putting in a couple of new features. Back then,
>> some of those new features (FLIP-6, network stack changes, local state
>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>> development cycle to allow for those features to become ready and then do
>> the next release.
>>
>> We are now approaching the approximate time where we wanted to do the
>> Flink 1.5.0 release so I would like to gauge where we are and gather
>> opinions on how we should proceed now.
>>
>> With this, I'd also like to propose myself as the release manager for
>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>> doing that.
>>
>> What do you think?
>>
>> Best,
>> Aljoscha
>>
>
>
>

--
Christophe

[Shuyi Chen]

Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
internal users waiting for this feature.

[FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
accessing subfields of a Composite element in an Object Array type column

Thanks a lot
Shuyi


On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]> wrote:

> Hi guys,
>
> Sorry for jumping in, but I think
>
> [FLINK-8101] Elasticsearch 6.X support
> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
> Elasticsearch 5.2+ client
>
>  have long been awaited and there was one PR from me and from someone else
> showing the interest ;) So if you could consider it for 1.5 that would be
> great!
>
> Thanks!
> --
> Christophe
>
> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]> wrote:
>
> > Hi Aljoscha,
> >
> > it would be great if we can include the first version of the SQL client
> > (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
> > we can merge this with explicit "experimental/alpha" status. It is far
> away
> > from feature completeness but will be a great tool for Flink beginners.
> >
> > In order to use the SQL client we would need to also add some table
> > sources with the new unified table factories (FLINK-8535), but this is
> > optional because a user can implement own table factories at the
> begining.
> >
> > Regards,
> > Timo
> >
> >
> > Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >
> > Hi Aljoscha,
> >>
> >> Thanks for starting the discussion.
> >>
> >> I think there’s a few connector related must-have improvements that we
> >> should get in before the feature freeze, since quite a few users have
> been
> >> asking for them:
> >>
> >> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
> up
> >> start offset
> >> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
> >> consider idle partitions
> >> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >> FlinkKinesisConsumer
> >> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
> >>
> >> These are still missing in the master branch. Only FLINK-5479 is still
> >> lacking a pull request.
> >>
> >> Cheers,
> >> Gordon
> >>
> >> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> [hidden email])
> >> wrote:
> >> Hi Everyone,
> >>
> >> When we decided to do the 1.4.0 release a while back we did that to get
> a
> >> stable release out before putting in a couple of new features. Back
> then,
> >> some of those new features (FLIP-6, network stack changes, local state
> >> recovery) were almost ready and we wanted to do a shortened 1.5.0
> >> development cycle to allow for those features to become ready and then
> do
> >> the next release.
> >>
> >> We are now approaching the approximate time where we wanted to do the
> >> Flink 1.5.0 release so I would like to gauge where we are and gather
> >> opinions on how we should proceed now.
> >>
> >> With this, I'd also like to propose myself as the release manager for
> >> 1.5.0 but I'm very happy to yield if someone else would be interested in
> >> doing that.
> >>
> >> What do you think?
> >>
> >> Best,
> >> Aljoscha
> >>
> >
> >
> >
>
>
> --
> Christophe
>

--
"So you have to trust that the dots will somehow connect in your future."

[Timo Walther-2]

Hi Shuyi,

I will take a look at it again this week. I'm pretty sure it will be
part of 1.5.0.

Regards,
Timo


Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:

> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
> internal users waiting for this feature.
>
> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
> accessing subfields of a Composite element in an Object Array type column
>
> Thanks a lot
> Shuyi
>
>
> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]> wrote:
>
>> Hi guys,
>>
>> Sorry for jumping in, but I think
>>
>> [FLINK-8101] Elasticsearch 6.X support
>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>> Elasticsearch 5.2+ client
>>
>>   have long been awaited and there was one PR from me and from someone else
>> showing the interest ;) So if you could consider it for 1.5 that would be
>> great!
>>
>> Thanks!
>> --
>> Christophe
>>
>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]> wrote:
>>
>>> Hi Aljoscha,
>>>
>>> it would be great if we can include the first version of the SQL client
>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
>>> we can merge this with explicit "experimental/alpha" status. It is far
>> away
>>> from feature completeness but will be a great tool for Flink beginners.
>>>
>>> In order to use the SQL client we would need to also add some table
>>> sources with the new unified table factories (FLINK-8535), but this is
>>> optional because a user can implement own table factories at the
>> begining.
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>
>>> Hi Aljoscha,
>>>> Thanks for starting the discussion.
>>>>
>>>> I think there’s a few connector related must-have improvements that we
>>>> should get in before the feature freeze, since quite a few users have
>> been
>>>> asking for them:
>>>>
>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>> up
>>>> start offset
>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>> consider idle partitions
>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>> FlinkKinesisConsumer
>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>
>>>> These are still missing in the master branch. Only FLINK-5479 is still
>>>> lacking a pull request.
>>>>
>>>> Cheers,
>>>> Gordon
>>>>
>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>> [hidden email])
>>>> wrote:
>>>> Hi Everyone,
>>>>
>>>> When we decided to do the 1.4.0 release a while back we did that to get
>> a
>>>> stable release out before putting in a couple of new features. Back
>> then,
>>>> some of those new features (FLIP-6, network stack changes, local state
>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>> development cycle to allow for those features to become ready and then
>> do
>>>> the next release.
>>>>
>>>> We are now approaching the approximate time where we wanted to do the
>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>> opinions on how we should proceed now.
>>>>
>>>> With this, I'd also like to propose myself as the release manager for
>>>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>>>> doing that.
>>>>
>>>> What do you think?
>>>>
>>>> Best,
>>>> Aljoscha
>>>>
>>>
>>>
>>
>> --
>> Christophe
>>
>
>

[Kostas Kloudas]

Hi Aljoscha,

I believe that support for Broadcast State should also be in 1.5.
There is an open PR https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230> for that
and there are some pending issues related to scala api and documentation.

Thanks,
Kostas

> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>
> Hi Shuyi,
>
> I will take a look at it again this week. I'm pretty sure it will be part of 1.5.0.
>
> Regards,
> Timo
>
>
> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>> internal users waiting for this feature.
>>
>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>> accessing subfields of a Composite element in an Object Array type column
>>
>> Thanks a lot
>> Shuyi
>>
>>
>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]> wrote:
>>
>>> Hi guys,
>>>
>>> Sorry for jumping in, but I think
>>>
>>> [FLINK-8101] Elasticsearch 6.X support
>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>> Elasticsearch 5.2+ client
>>>
>>>  have long been awaited and there was one PR from me and from someone else
>>> showing the interest ;) So if you could consider it for 1.5 that would be
>>> great!
>>>
>>> Thanks!
>>> --
>>> Christophe
>>>
>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]> wrote:
>>>
>>>> Hi Aljoscha,
>>>>
>>>> it would be great if we can include the first version of the SQL client
>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>> away
>>>> from feature completeness but will be a great tool for Flink beginners.
>>>>
>>>> In order to use the SQL client we would need to also add some table
>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>> optional because a user can implement own table factories at the
>>> begining.
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>
>>>> Hi Aljoscha,
>>>>> Thanks for starting the discussion.
>>>>>
>>>>> I think there’s a few connector related must-have improvements that we
>>>>> should get in before the feature freeze, since quite a few users have
>>> been
>>>>> asking for them:
>>>>>
>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>>> up
>>>>> start offset
>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>> consider idle partitions
>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>> FlinkKinesisConsumer
>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>
>>>>> These are still missing in the master branch. Only FLINK-5479 is still
>>>>> lacking a pull request.
>>>>>
>>>>> Cheers,
>>>>> Gordon
>>>>>
>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>> [hidden email])
>>>>> wrote:
>>>>> Hi Everyone,
>>>>>
>>>>> When we decided to do the 1.4.0 release a while back we did that to get
>>> a
>>>>> stable release out before putting in a couple of new features. Back
>>> then,
>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>> development cycle to allow for those features to become ready and then
>>> do
>>>>> the next release.
>>>>>
>>>>> We are now approaching the approximate time where we wanted to do the
>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>> opinions on how we should proceed now.
>>>>>
>>>>> With this, I'd also like to propose myself as the release manager for
>>>>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>>>>> doing that.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Best,
>>>>> Aljoscha
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Christophe
>>>
>>
>>
>

[Chesnay Schepler-3]

As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
assumption that the 3 big features (FLIP-6, network stack changes, local
state recovery) are nearly done.

I'm unsure about local state recovery, but I still see open issues for
FLIP-6 and the network stack rework.
As such it doesn't make sense to release 1.5 now.

Given the large scope of these features I would very much prefer to have
them active on master for a while before a feature-freeze
to expose them to a wider audience.

IMO it will take at least another month before we can start the release
process for 1.5, i.e. the feature freeze.
(2 more weeks for implementation, 2 weeks on master for the dust to settle)

On 05.02.2018 22:39, Kostas Kloudas wrote:

> Hi Aljoscha,
>
> I believe that support for Broadcast State should also be in 1.5.
> There is an open PR https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230> for that
> and there are some pending issues related to scala api and documentation.
>
> Thanks,
> Kostas
>
>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>>
>> Hi Shuyi,
>>
>> I will take a look at it again this week. I'm pretty sure it will be part of 1.5.0.
>>
>> Regards,
>> Timo
>>
>>
>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>> internal users waiting for this feature.
>>>
>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>> accessing subfields of a Composite element in an Object Array type column
>>>
>>> Thanks a lot
>>> Shuyi
>>>
>>>
>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> Sorry for jumping in, but I think
>>>>
>>>> [FLINK-8101] Elasticsearch 6.X support
>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>>> Elasticsearch 5.2+ client
>>>>
>>>>   have long been awaited and there was one PR from me and from someone else
>>>> showing the interest ;) So if you could consider it for 1.5 that would be
>>>> great!
>>>>
>>>> Thanks!
>>>> --
>>>> Christophe
>>>>
>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]> wrote:
>>>>
>>>>> Hi Aljoscha,
>>>>>
>>>>> it would be great if we can include the first version of the SQL client
>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I think
>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>> away
>>>>> from feature completeness but will be a great tool for Flink beginners.
>>>>>
>>>>> In order to use the SQL client we would need to also add some table
>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>> optional because a user can implement own table factories at the
>>>> begining.
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>
>>>>> Hi Aljoscha,
>>>>>> Thanks for starting the discussion.
>>>>>>
>>>>>> I think there’s a few connector related must-have improvements that we
>>>>>> should get in before the feature freeze, since quite a few users have
>>>> been
>>>>>> asking for them:
>>>>>>
>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to set
>>>> up
>>>>>> start offset
>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>> consider idle partitions
>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>> FlinkKinesisConsumer
>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>
>>>>>> These are still missing in the master branch. Only FLINK-5479 is still
>>>>>> lacking a pull request.
>>>>>>
>>>>>> Cheers,
>>>>>> Gordon
>>>>>>
>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>> [hidden email])
>>>>>> wrote:
>>>>>> Hi Everyone,
>>>>>>
>>>>>> When we decided to do the 1.4.0 release a while back we did that to get
>>>> a
>>>>>> stable release out before putting in a couple of new features. Back
>>>> then,
>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>> development cycle to allow for those features to become ready and then
>>>> do
>>>>>> the next release.
>>>>>>
>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>> opinions on how we should proceed now.
>>>>>>
>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>> 1.5.0 but I'm very happy to yield if someone else would be interested in
>>>>>> doing that.
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> Best,
>>>>>> Aljoscha
>>>>>>
>>>>>
>>>> --
>>>> Christophe
>>>>
>>>
>

[Till Rohrmann]

Local state recovery is almost completely done. Only some reviews and
merging of the final PRs is pending.

The network stack improvements are on a good way to be finished by the end
of this week or beginning of next week. To my knowledge we got recently
green Travis builds :-) The network stack changes will also include the
application level flow control and the back pressure based checkpoint
alignment. So only the last reviews and merging is missing.

Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
There are still some smaller things left to be done but I'm confident that
we can resolve them quickly.

I agree that due to the big changes we should have a very thorough and
principled testing period where we put Flink through the paces.

Cheers,
Till

On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <[hidden email]>
wrote:

> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> assumption that the 3 big features (FLIP-6, network stack changes, local
> state recovery) are nearly done.
>
> I'm unsure about local state recovery, but I still see open issues for
> FLIP-6 and the network stack rework.
> As such it doesn't make sense to release 1.5 now.
>
> Given the large scope of these features I would very much prefer to have
> them active on master for a while before a feature-freeze
> to expose them to a wider audience.
>
> IMO it will take at least another month before we can start the release
> process for 1.5, i.e. the feature freeze.
> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>
>
> On 05.02.2018 22:39, Kostas Kloudas wrote:
>
>> Hi Aljoscha,
>>
>> I believe that support for Broadcast State should also be in 1.5.
>> There is an open PR https://github.com/apache/flink/pull/5230 <
>> https://github.com/apache/flink/pull/5230> for that
>> and there are some pending issues related to scala api and documentation.
>>
>> Thanks,
>> Kostas
>>
>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>>>
>>> Hi Shuyi,
>>>
>>> I will take a look at it again this week. I'm pretty sure it will be
>>> part of 1.5.0.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>
>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>> internal users waiting for this feature.
>>>>
>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>> accessing subfields of a Composite element in an Object Array type
>>>> column
>>>>
>>>> Thanks a lot
>>>> Shuyi
>>>>
>>>>
>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]>
>>>> wrote:
>>>>
>>>> Hi guys,
>>>>>
>>>>> Sorry for jumping in, but I think
>>>>>
>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>>>> Elasticsearch 5.2+ client
>>>>>
>>>>>   have long been awaited and there was one PR from me and from someone
>>>>> else
>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>> be
>>>>> great!
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> Christophe
>>>>>
>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Hi Aljoscha,
>>>>>>
>>>>>> it would be great if we can include the first version of the SQL
>>>>>> client
>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>> think
>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>
>>>>> away
>>>>>
>>>>>> from feature completeness but will be a great tool for Flink
>>>>>> beginners.
>>>>>>
>>>>>> In order to use the SQL client we would need to also add some table
>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>> optional because a user can implement own table factories at the
>>>>>>
>>>>> begining.
>>>>>
>>>>>> Regards,
>>>>>> Timo
>>>>>>
>>>>>>
>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>
>>>>>> Hi Aljoscha,
>>>>>>
>>>>>>> Thanks for starting the discussion.
>>>>>>>
>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>> we
>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>
>>>>>> been
>>>>>
>>>>>> asking for them:
>>>>>>>
>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>> set
>>>>>>>
>>>>>> up
>>>>>
>>>>>> start offset
>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>> consider idle partitions
>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>> FlinkKinesisConsumer
>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>
>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>> still
>>>>>>> lacking a pull request.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Gordon
>>>>>>>
>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>
>>>>>> [hidden email])
>>>>>
>>>>>> wrote:
>>>>>>> Hi Everyone,
>>>>>>>
>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>> get
>>>>>>>
>>>>>> a
>>>>>
>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>
>>>>>> then,
>>>>>
>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>> development cycle to allow for those features to become ready and
>>>>>>> then
>>>>>>>
>>>>>> do
>>>>>
>>>>>> the next release.
>>>>>>>
>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>> opinions on how we should proceed now.
>>>>>>>
>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>> interested in
>>>>>>> doing that.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Best,
>>>>>>> Aljoscha
>>>>>>>
>>>>>>>
>>>>>> --
>>>>> Christophe
>>>>>
>>>>>
>>>>
>>
>

[Aljoscha Krettek-2]

I agree with Chesnay: we should do a soft "feature freeze" first, were we agree to not merge new features to master after that and then to the actual hard cutting of the release branch a while later.

For actual dates, I'm proposing end of this week (16.02.2018) as soft feature freeze and end of next week (23.02.2018) as the hard cut of the release branch?

What do you think?

--
Aljoscha

> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]> wrote:
>
> Local state recovery is almost completely done. Only some reviews and
> merging of the final PRs is pending.
>
> The network stack improvements are on a good way to be finished by the end
> of this week or beginning of next week. To my knowledge we got recently
> green Travis builds :-) The network stack changes will also include the
> application level flow control and the back pressure based checkpoint
> alignment. So only the last reviews and merging is missing.
>
> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
> There are still some smaller things left to be done but I'm confident that
> we can resolve them quickly.
>
> I agree that due to the big changes we should have a very thorough and
> principled testing period where we put Flink through the paces.
>
> Cheers,
> Till
>
> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <[hidden email]>
> wrote:
>
>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>> assumption that the 3 big features (FLIP-6, network stack changes, local
>> state recovery) are nearly done.
>>
>> I'm unsure about local state recovery, but I still see open issues for
>> FLIP-6 and the network stack rework.
>> As such it doesn't make sense to release 1.5 now.
>>
>> Given the large scope of these features I would very much prefer to have
>> them active on master for a while before a feature-freeze
>> to expose them to a wider audience.
>>
>> IMO it will take at least another month before we can start the release
>> process for 1.5, i.e. the feature freeze.
>> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>>
>>
>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>
>>> Hi Aljoscha,
>>>
>>> I believe that support for Broadcast State should also be in 1.5.
>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>> https://github.com/apache/flink/pull/5230> for that
>>> and there are some pending issues related to scala api and documentation.
>>>
>>> Thanks,
>>> Kostas
>>>
>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>>>>
>>>> Hi Shuyi,
>>>>
>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>> part of 1.5.0.
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>
>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>> internal users waiting for this feature.
>>>>>
>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>>> accessing subfields of a Composite element in an Object Array type
>>>>> column
>>>>>
>>>>> Thanks a lot
>>>>> Shuyi
>>>>>
>>>>>
>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]>
>>>>> wrote:
>>>>>
>>>>> Hi guys,
>>>>>>
>>>>>> Sorry for jumping in, but I think
>>>>>>
>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>>>>> Elasticsearch 5.2+ client
>>>>>>
>>>>>>  have long been awaited and there was one PR from me and from someone
>>>>>> else
>>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>>> be
>>>>>> great!
>>>>>>
>>>>>> Thanks!
>>>>>> --
>>>>>> Christophe
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Aljoscha,
>>>>>>>
>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>> client
>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>>> think
>>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>>
>>>>>> away
>>>>>>
>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>> beginners.
>>>>>>>
>>>>>>> In order to use the SQL client we would need to also add some table
>>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>>> optional because a user can implement own table factories at the
>>>>>>>
>>>>>> begining.
>>>>>>
>>>>>>> Regards,
>>>>>>> Timo
>>>>>>>
>>>>>>>
>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>
>>>>>>> Hi Aljoscha,
>>>>>>>
>>>>>>>> Thanks for starting the discussion.
>>>>>>>>
>>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>>> we
>>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>>
>>>>>>> been
>>>>>>
>>>>>>> asking for them:
>>>>>>>>
>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>>> set
>>>>>>>>
>>>>>>> up
>>>>>>
>>>>>>> start offset
>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>>> consider idle partitions
>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>> FlinkKinesisConsumer
>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>>
>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>> still
>>>>>>>> lacking a pull request.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Gordon
>>>>>>>>
>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>
>>>>>>> [hidden email])
>>>>>>
>>>>>>> wrote:
>>>>>>>> Hi Everyone,
>>>>>>>>
>>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>>> get
>>>>>>>>
>>>>>>> a
>>>>>>
>>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>>
>>>>>>> then,
>>>>>>
>>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>>> development cycle to allow for those features to become ready and
>>>>>>>> then
>>>>>>>>
>>>>>>> do
>>>>>>
>>>>>>> the next release.
>>>>>>>>
>>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>>> opinions on how we should proceed now.
>>>>>>>>
>>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>> interested in
>>>>>>>> doing that.
>>>>>>>>
>>>>>>>> What do you think?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Aljoscha
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>> Christophe
>>>>>>
>>>>>>
>>>>>
>>>
>>

[Timo Walther-2]

Sounds good to me. +1 from my side.

Regards,
Timo


Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:

> I agree with Chesnay: we should do a soft "feature freeze" first, were we agree to not merge new features to master after that and then to the actual hard cutting of the release branch a while later.
>
> For actual dates, I'm proposing end of this week (16.02.2018) as soft feature freeze and end of next week (23.02.2018) as the hard cut of the release branch?
>
> What do you think?
>
> --
> Aljoscha
>
>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]> wrote:
>>
>> Local state recovery is almost completely done. Only some reviews and
>> merging of the final PRs is pending.
>>
>> The network stack improvements are on a good way to be finished by the end
>> of this week or beginning of next week. To my knowledge we got recently
>> green Travis builds :-) The network stack changes will also include the
>> application level flow control and the back pressure based checkpoint
>> alignment. So only the last reviews and merging is missing.
>>
>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
>> There are still some smaller things left to be done but I'm confident that
>> we can resolve them quickly.
>>
>> I agree that due to the big changes we should have a very thorough and
>> principled testing period where we put Flink through the paces.
>>
>> Cheers,
>> Till
>>
>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <[hidden email]>
>> wrote:
>>
>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>> assumption that the 3 big features (FLIP-6, network stack changes, local
>>> state recovery) are nearly done.
>>>
>>> I'm unsure about local state recovery, but I still see open issues for
>>> FLIP-6 and the network stack rework.
>>> As such it doesn't make sense to release 1.5 now.
>>>
>>> Given the large scope of these features I would very much prefer to have
>>> them active on master for a while before a feature-freeze
>>> to expose them to a wider audience.
>>>
>>> IMO it will take at least another month before we can start the release
>>> process for 1.5, i.e. the feature freeze.
>>> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>>>
>>>
>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>
>>>> Hi Aljoscha,
>>>>
>>>> I believe that support for Broadcast State should also be in 1.5.
>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>> https://github.com/apache/flink/pull/5230> for that
>>>> and there are some pending issues related to scala api and documentation.
>>>>
>>>> Thanks,
>>>> Kostas
>>>>
>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>>>>> Hi Shuyi,
>>>>>
>>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>>> part of 1.5.0.
>>>>>
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>
>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>>> internal users waiting for this feature.
>>>>>>
>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>>>> accessing subfields of a Composite element in an Object Array type
>>>>>> column
>>>>>>
>>>>>> Thanks a lot
>>>>>> Shuyi
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>>> Sorry for jumping in, but I think
>>>>>>>
>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>>>>>> Elasticsearch 5.2+ client
>>>>>>>
>>>>>>>   have long been awaited and there was one PR from me and from someone
>>>>>>> else
>>>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>>>> be
>>>>>>> great!
>>>>>>>
>>>>>>> Thanks!
>>>>>>> --
>>>>>>> Christophe
>>>>>>>
>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Aljoscha,
>>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>>> client
>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>>>> think
>>>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>>>
>>>>>>> away
>>>>>>>
>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>> beginners.
>>>>>>>>
>>>>>>>> In order to use the SQL client we would need to also add some table
>>>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>>>> optional because a user can implement own table factories at the
>>>>>>>>
>>>>>>> begining.
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Timo
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>
>>>>>>>> Hi Aljoscha,
>>>>>>>>
>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>
>>>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>>>> we
>>>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>>>
>>>>>>>> been
>>>>>>>> asking for them:
>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>>>> set
>>>>>>>>>
>>>>>>>> up
>>>>>>>> start offset
>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>>>> consider idle partitions
>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>>>
>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>>> still
>>>>>>>>> lacking a pull request.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Gordon
>>>>>>>>>
>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>
>>>>>>>> [hidden email])
>>>>>>>> wrote:
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>>>> get
>>>>>>>>>
>>>>>>>> a
>>>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>> then,
>>>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>>>> development cycle to allow for those features to become ready and
>>>>>>>>> then
>>>>>>>>>
>>>>>>>> do
>>>>>>>> the next release.
>>>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>
>>>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>> interested in
>>>>>>>>> doing that.
>>>>>>>>>
>>>>>>>>> What do you think?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Aljoscha
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>> Christophe
>>>>>>>
>>>>>>>

[Kostas Kloudas]

For me as well +1.

Cheers,
Kostas

> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]> wrote:
>
> Sounds good to me. +1 from my side.
>
> Regards,
> Timo
>
>
> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>> I agree with Chesnay: we should do a soft "feature freeze" first, were we agree to not merge new features to master after that and then to the actual hard cutting of the release branch a while later.
>>
>> For actual dates, I'm proposing end of this week (16.02.2018) as soft feature freeze and end of next week (23.02.2018) as the hard cut of the release branch?
>>
>> What do you think?
>>
>> --
>> Aljoscha
>>
>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]> wrote:
>>>
>>> Local state recovery is almost completely done. Only some reviews and
>>> merging of the final PRs is pending.
>>>
>>> The network stack improvements are on a good way to be finished by the end
>>> of this week or beginning of next week. To my knowledge we got recently
>>> green Travis builds :-) The network stack changes will also include the
>>> application level flow control and the back pressure based checkpoint
>>> alignment. So only the last reviews and merging is missing.
>>>
>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
>>> There are still some smaller things left to be done but I'm confident that
>>> we can resolve them quickly.
>>>
>>> I agree that due to the big changes we should have a very thorough and
>>> principled testing period where we put Flink through the paces.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <[hidden email]>
>>> wrote:
>>>
>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>>> assumption that the 3 big features (FLIP-6, network stack changes, local
>>>> state recovery) are nearly done.
>>>>
>>>> I'm unsure about local state recovery, but I still see open issues for
>>>> FLIP-6 and the network stack rework.
>>>> As such it doesn't make sense to release 1.5 now.
>>>>
>>>> Given the large scope of these features I would very much prefer to have
>>>> them active on master for a while before a feature-freeze
>>>> to expose them to a wider audience.
>>>>
>>>> IMO it will take at least another month before we can start the release
>>>> process for 1.5, i.e. the feature freeze.
>>>> (2 more weeks for implementation, 2 weeks on master for the dust to settle)
>>>>
>>>>
>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>>
>>>>> Hi Aljoscha,
>>>>>
>>>>> I believe that support for Broadcast State should also be in 1.5.
>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>>> https://github.com/apache/flink/pull/5230> for that
>>>>> and there are some pending issues related to scala api and documentation.
>>>>>
>>>>> Thanks,
>>>>> Kostas
>>>>>
>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>>>>>> Hi Shuyi,
>>>>>>
>>>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>>>> part of 1.5.0.
>>>>>>
>>>>>> Regards,
>>>>>> Timo
>>>>>>
>>>>>>
>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>>
>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>>>> internal users waiting for this feature.
>>>>>>>
>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>] Support
>>>>>>> accessing subfields of a Composite element in an Object Array type
>>>>>>> column
>>>>>>>
>>>>>>> Thanks a lot
>>>>>>> Shuyi
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi guys,
>>>>>>>> Sorry for jumping in, but I think
>>>>>>>>
>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible with
>>>>>>>> Elasticsearch 5.2+ client
>>>>>>>>
>>>>>>>>  have long been awaited and there was one PR from me and from someone
>>>>>>>> else
>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that would
>>>>>>>> be
>>>>>>>> great!
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> --
>>>>>>>> Christophe
>>>>>>>>
>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Aljoscha,
>>>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>>>> client
>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this week. I
>>>>>>>>> think
>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It is far
>>>>>>>>>
>>>>>>>> away
>>>>>>>>
>>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>>> beginners.
>>>>>>>>>
>>>>>>>>> In order to use the SQL client we would need to also add some table
>>>>>>>>> sources with the new unified table factories (FLINK-8535), but this is
>>>>>>>>> optional because a user can implement own table factories at the
>>>>>>>>>
>>>>>>>> begining.
>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Timo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>>
>>>>>>>>> Hi Aljoscha,
>>>>>>>>>
>>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>>
>>>>>>>>>> I think there’s a few connector related must-have improvements that
>>>>>>>>>> we
>>>>>>>>>> should get in before the feature freeze, since quite a few users have
>>>>>>>>>>
>>>>>>>>> been
>>>>>>>>> asking for them:
>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp to
>>>>>>>>>> set
>>>>>>>>>>
>>>>>>>>> up
>>>>>>>>> start offset
>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer should
>>>>>>>>>> consider idle partitions
>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to FlinkKafkaConsumer
>>>>>>>>>>
>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>>>> still
>>>>>>>>>> lacking a pull request.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Gordon
>>>>>>>>>>
>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>>
>>>>>>>>> [hidden email])
>>>>>>>>> wrote:
>>>>>>>>>> Hi Everyone,
>>>>>>>>>>
>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did that to
>>>>>>>>>> get
>>>>>>>>>>
>>>>>>>>> a
>>>>>>>>> stable release out before putting in a couple of new features. Back
>>>>>>>>> then,
>>>>>>>>> some of those new features (FLIP-6, network stack changes, local state
>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened 1.5.0
>>>>>>>>>> development cycle to allow for those features to become ready and
>>>>>>>>>> then
>>>>>>>>>>
>>>>>>>>> do
>>>>>>>>> the next release.
>>>>>>>>>> We are now approaching the approximate time where we wanted to do the
>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and gather
>>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>>
>>>>>>>>>> With this, I'd also like to propose myself as the release manager for
>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>>> interested in
>>>>>>>>>> doing that.
>>>>>>>>>>
>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Aljoscha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Christophe
>>>>>>>>
>>>>>>>>
>

[Stephan Ewen]

I agree with the basic idea. I think there is no need to call it "soft
feature freeze" though - it is a feature freeze (no new features get
merged) ;-)

What you are suggesting is to delay forking of the release-1.5 branch to
avoid applying the bug fixes to too many branches. That makes sense.
In effect, there is a period of one week (next week) where no post 1.5
features can be merged (feature freeze but release branch not forked),
which should be okay.

Best,
Stephan


On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <[hidden email]
> wrote:

> For me as well +1.
>
> Cheers,
> Kostas
>
> > On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]> wrote:
> >
> > Sounds good to me. +1 from my side.
> >
> > Regards,
> > Timo
> >
> >
> > Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> >> I agree with Chesnay: we should do a soft "feature freeze" first, were
> we agree to not merge new features to master after that and then to the
> actual hard cutting of the release branch a while later.
> >>
> >> For actual dates, I'm proposing end of this week (16.02.2018) as soft
> feature freeze and end of next week (23.02.2018) as the hard cut of the
> release branch?
> >>
> >> What do you think?
> >>
> >> --
> >> Aljoscha
> >>
> >>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]> wrote:
> >>>
> >>> Local state recovery is almost completely done. Only some reviews and
> >>> merging of the final PRs is pending.
> >>>
> >>> The network stack improvements are on a good way to be finished by the
> end
> >>> of this week or beginning of next week. To my knowledge we got recently
> >>> green Travis builds :-) The network stack changes will also include the
> >>> application level flow control and the back pressure based checkpoint
> >>> alignment. So only the last reviews and merging is missing.
> >>>
> >>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
> >>> There are still some smaller things left to be done but I'm confident
> that
> >>> we can resolve them quickly.
> >>>
> >>> I agree that due to the big changes we should have a very thorough and
> >>> principled testing period where we put Flink through the paces.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <[hidden email]>
> >>> wrote:
> >>>
> >>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> >>>> assumption that the 3 big features (FLIP-6, network stack changes,
> local
> >>>> state recovery) are nearly done.
> >>>>
> >>>> I'm unsure about local state recovery, but I still see open issues for
> >>>> FLIP-6 and the network stack rework.
> >>>> As such it doesn't make sense to release 1.5 now.
> >>>>
> >>>> Given the large scope of these features I would very much prefer to
> have
> >>>> them active on master for a while before a feature-freeze
> >>>> to expose them to a wider audience.
> >>>>
> >>>> IMO it will take at least another month before we can start the
> release
> >>>> process for 1.5, i.e. the feature freeze.
> >>>> (2 more weeks for implementation, 2 weeks on master for the dust to
> settle)
> >>>>
> >>>>
> >>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> >>>>
> >>>>> Hi Aljoscha,
> >>>>>
> >>>>> I believe that support for Broadcast State should also be in 1.5.
> >>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
> >>>>> https://github.com/apache/flink/pull/5230> for that
> >>>>> and there are some pending issues related to scala api and
> documentation.
> >>>>>
> >>>>> Thanks,
> >>>>> Kostas
> >>>>>
> >>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
> >>>>>> Hi Shuyi,
> >>>>>>
> >>>>>> I will take a look at it again this week. I'm pretty sure it will be
> >>>>>> part of 1.5.0.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Timo
> >>>>>>
> >>>>>>
> >>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> >>>>>>
> >>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
> >>>>>>> internal users waiting for this feature.
> >>>>>>>
> >>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
> Support
> >>>>>>> accessing subfields of a Composite element in an Object Array type
> >>>>>>> column
> >>>>>>>
> >>>>>>> Thanks a lot
> >>>>>>> Shuyi
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]
> >
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi guys,
> >>>>>>>> Sorry for jumping in, but I think
> >>>>>>>>
> >>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> >>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible
> with
> >>>>>>>> Elasticsearch 5.2+ client
> >>>>>>>>
> >>>>>>>>  have long been awaited and there was one PR from me and from
> someone
> >>>>>>>> else
> >>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
> would
> >>>>>>>> be
> >>>>>>>> great!
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>> --
> >>>>>>>> Christophe
> >>>>>>>>
> >>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Aljoscha,
> >>>>>>>>> it would be great if we can include the first version of the SQL
> >>>>>>>>> client
> >>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> week. I
> >>>>>>>>> think
> >>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
> is far
> >>>>>>>>>
> >>>>>>>> away
> >>>>>>>>
> >>>>>>>>> from feature completeness but will be a great tool for Flink
> >>>>>>>>> beginners.
> >>>>>>>>>
> >>>>>>>>> In order to use the SQL client we would need to also add some
> table
> >>>>>>>>> sources with the new unified table factories (FLINK-8535), but
> this is
> >>>>>>>>> optional because a user can implement own table factories at the
> >>>>>>>>>
> >>>>>>>> begining.
> >>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Timo
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >>>>>>>>>
> >>>>>>>>> Hi Aljoscha,
> >>>>>>>>>
> >>>>>>>>>> Thanks for starting the discussion.
> >>>>>>>>>>
> >>>>>>>>>> I think there’s a few connector related must-have improvements
> that
> >>>>>>>>>> we
> >>>>>>>>>> should get in before the feature freeze, since quite a few
> users have
> >>>>>>>>>>
> >>>>>>>>> been
> >>>>>>>>> asking for them:
> >>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp
> to
> >>>>>>>>>> set
> >>>>>>>>>>
> >>>>>>>>> up
> >>>>>>>>> start offset
> >>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> should
> >>>>>>>>>> consider idle partitions
> >>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >>>>>>>>>> FlinkKinesisConsumer
> >>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> FlinkKafkaConsumer
> >>>>>>>>>>
> >>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
> >>>>>>>>>> still
> >>>>>>>>>> lacking a pull request.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Gordon
> >>>>>>>>>>
> >>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> >>>>>>>>>>
> >>>>>>>>> [hidden email])
> >>>>>>>>> wrote:
> >>>>>>>>>> Hi Everyone,
> >>>>>>>>>>
> >>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> that to
> >>>>>>>>>> get
> >>>>>>>>>>
> >>>>>>>>> a
> >>>>>>>>> stable release out before putting in a couple of new features.
> Back
> >>>>>>>>> then,
> >>>>>>>>> some of those new features (FLIP-6, network stack changes, local
> state
> >>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> 1.5.0
> >>>>>>>>>> development cycle to allow for those features to become ready
> and
> >>>>>>>>>> then
> >>>>>>>>>>
> >>>>>>>>> do
> >>>>>>>>> the next release.
> >>>>>>>>>> We are now approaching the approximate time where we wanted to
> do the
> >>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
> gather
> >>>>>>>>>> opinions on how we should proceed now.
> >>>>>>>>>>
> >>>>>>>>>> With this, I'd also like to propose myself as the release
> manager for
> >>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> >>>>>>>>>> interested in
> >>>>>>>>>> doing that.
> >>>>>>>>>>
> >>>>>>>>>> What do you think?
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Aljoscha
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>> Christophe
> >>>>>>>>
> >>>>>>>>
> >
>
>

[Piotr Nowojski]

+0.95 from my side.

Network changes are mostly reviewed and should be merged by the end of this week.

Piotrek

> On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email]> wrote:
>
> I agree with the basic idea. I think there is no need to call it "soft
> feature freeze" though - it is a feature freeze (no new features get
> merged) ;-)
>
> What you are suggesting is to delay forking of the release-1.5 branch to
> avoid applying the bug fixes to too many branches. That makes sense.
> In effect, there is a period of one week (next week) where no post 1.5
> features can be merged (feature freeze but release branch not forked),
> which should be okay.
>
> Best,
> Stephan
>
>
> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <[hidden email]
>> wrote:
>
>> For me as well +1.
>>
>> Cheers,
>> Kostas
>>
>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]> wrote:
>>>
>>> Sounds good to me. +1 from my side.
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>>>> I agree with Chesnay: we should do a soft "feature freeze" first, were
>> we agree to not merge new features to master after that and then to the
>> actual hard cutting of the release branch a while later.
>>>>
>>>> For actual dates, I'm proposing end of this week (16.02.2018) as soft
>> feature freeze and end of next week (23.02.2018) as the hard cut of the
>> release branch?
>>>>
>>>> What do you think?
>>>>
>>>> --
>>>> Aljoscha
>>>>
>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]> wrote:
>>>>>
>>>>> Local state recovery is almost completely done. Only some reviews and
>>>>> merging of the final PRs is pending.
>>>>>
>>>>> The network stack improvements are on a good way to be finished by the
>> end
>>>>> of this week or beginning of next week. To my knowledge we got recently
>>>>> green Travis builds :-) The network stack changes will also include the
>>>>> application level flow control and the back pressure based checkpoint
>>>>> alignment. So only the last reviews and merging is missing.
>>>>>
>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by default.
>>>>> There are still some smaller things left to be done but I'm confident
>> that
>>>>> we can resolve them quickly.
>>>>>
>>>>> I agree that due to the big changes we should have a very thorough and
>>>>> principled testing period where we put Flink through the paces.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
>> local
>>>>>> state recovery) are nearly done.
>>>>>>
>>>>>> I'm unsure about local state recovery, but I still see open issues for
>>>>>> FLIP-6 and the network stack rework.
>>>>>> As such it doesn't make sense to release 1.5 now.
>>>>>>
>>>>>> Given the large scope of these features I would very much prefer to
>> have
>>>>>> them active on master for a while before a feature-freeze
>>>>>> to expose them to a wider audience.
>>>>>>
>>>>>> IMO it will take at least another month before we can start the
>> release
>>>>>> process for 1.5, i.e. the feature freeze.
>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust to
>> settle)
>>>>>>
>>>>>>
>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>>>>
>>>>>>> Hi Aljoscha,
>>>>>>>
>>>>>>> I believe that support for Broadcast State should also be in 1.5.
>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>>>>> https://github.com/apache/flink/pull/5230> for that
>>>>>>> and there are some pending issues related to scala api and
>> documentation.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Kostas
>>>>>>>
>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]> wrote:
>>>>>>>> Hi Shuyi,
>>>>>>>>
>>>>>>>> I will take a look at it again this week. I'm pretty sure it will be
>>>>>>>> part of 1.5.0.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Timo
>>>>>>>>
>>>>>>>>
>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>>>>
>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot of
>>>>>>>>> internal users waiting for this feature.
>>>>>>>>>
>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
>> Support
>>>>>>>>> accessing subfields of a Composite element in an Object Array type
>>>>>>>>> column
>>>>>>>>>
>>>>>>>>> Thanks a lot
>>>>>>>>> Shuyi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <[hidden email]
>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi guys,
>>>>>>>>>> Sorry for jumping in, but I think
>>>>>>>>>>
>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible
>> with
>>>>>>>>>> Elasticsearch 5.2+ client
>>>>>>>>>>
>>>>>>>>>> have long been awaited and there was one PR from me and from
>> someone
>>>>>>>>>> else
>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
>> would
>>>>>>>>>> be
>>>>>>>>>> great!
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> --
>>>>>>>>>> Christophe
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <[hidden email]>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>> it would be great if we can include the first version of the SQL
>>>>>>>>>>> client
>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
>> week. I
>>>>>>>>>>> think
>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
>> is far
>>>>>>>>>>>
>>>>>>>>>> away
>>>>>>>>>>
>>>>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>>>>> beginners.
>>>>>>>>>>>
>>>>>>>>>>> In order to use the SQL client we would need to also add some
>> table
>>>>>>>>>>> sources with the new unified table factories (FLINK-8535), but
>> this is
>>>>>>>>>>> optional because a user can implement own table factories at the
>>>>>>>>>>>
>>>>>>>>>> begining.
>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Timo
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>>>>
>>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>>>>
>>>>>>>>>>>> I think there’s a few connector related must-have improvements
>> that
>>>>>>>>>>>> we
>>>>>>>>>>>> should get in before the feature freeze, since quite a few
>> users have
>>>>>>>>>>>>
>>>>>>>>>>> been
>>>>>>>>>>> asking for them:
>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use timestamp
>> to
>>>>>>>>>>>> set
>>>>>>>>>>>>
>>>>>>>>>>> up
>>>>>>>>>>> start offset
>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
>> should
>>>>>>>>>>>> consider idle partitions
>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
>> FlinkKafkaConsumer
>>>>>>>>>>>>
>>>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479 is
>>>>>>>>>>>> still
>>>>>>>>>>>> lacking a pull request.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Gordon
>>>>>>>>>>>>
>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>>>>
>>>>>>>>>>> [hidden email])
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
>> that to
>>>>>>>>>>>> get
>>>>>>>>>>>>
>>>>>>>>>>> a
>>>>>>>>>>> stable release out before putting in a couple of new features.
>> Back
>>>>>>>>>>> then,
>>>>>>>>>>> some of those new features (FLIP-6, network stack changes, local
>> state
>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
>> 1.5.0
>>>>>>>>>>>> development cycle to allow for those features to become ready
>> and
>>>>>>>>>>>> then
>>>>>>>>>>>>
>>>>>>>>>>> do
>>>>>>>>>>> the next release.
>>>>>>>>>>>> We are now approaching the approximate time where we wanted to
>> do the
>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
>> gather
>>>>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>>>>
>>>>>>>>>>>> With this, I'd also like to propose myself as the release
>> manager for
>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>>>>> interested in
>>>>>>>>>>>> doing that.
>>>>>>>>>>>>
>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Christophe
>>>>>>>>>>
>>>>>>>>>>
>>>
>>
>>

[Till Rohrmann]

+1 from my side.

Cheers,
Till

On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <[hidden email]>
wrote:

> +0.95 from my side.
>
> Network changes are mostly reviewed and should be merged by the end of
> this week.
>
> Piotrek
>
> > On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email]> wrote:
> >
> > I agree with the basic idea. I think there is no need to call it "soft
> > feature freeze" though - it is a feature freeze (no new features get
> > merged) ;-)
> >
> > What you are suggesting is to delay forking of the release-1.5 branch to
> > avoid applying the bug fixes to too many branches. That makes sense.
> > In effect, there is a period of one week (next week) where no post 1.5
> > features can be merged (feature freeze but release branch not forked),
> > which should be okay.
> >
> > Best,
> > Stephan
> >
> >
> > On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> [hidden email]
> >> wrote:
> >
> >> For me as well +1.
> >>
> >> Cheers,
> >> Kostas
> >>
> >>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]> wrote:
> >>>
> >>> Sounds good to me. +1 from my side.
> >>>
> >>> Regards,
> >>> Timo
> >>>
> >>>
> >>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> >>>> I agree with Chesnay: we should do a soft "feature freeze" first, were
> >> we agree to not merge new features to master after that and then to the
> >> actual hard cutting of the release branch a while later.
> >>>>
> >>>> For actual dates, I'm proposing end of this week (16.02.2018) as soft
> >> feature freeze and end of next week (23.02.2018) as the hard cut of the
> >> release branch?
> >>>>
> >>>> What do you think?
> >>>>
> >>>> --
> >>>> Aljoscha
> >>>>
> >>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]>
> wrote:
> >>>>>
> >>>>> Local state recovery is almost completely done. Only some reviews and
> >>>>> merging of the final PRs is pending.
> >>>>>
> >>>>> The network stack improvements are on a good way to be finished by
> the
> >> end
> >>>>> of this week or beginning of next week. To my knowledge we got
> recently
> >>>>> green Travis builds :-) The network stack changes will also include
> the
> >>>>> application level flow control and the back pressure based checkpoint
> >>>>> alignment. So only the last reviews and merging is missing.
> >>>>>
> >>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> default.
> >>>>> There are still some smaller things left to be done but I'm confident
> >> that
> >>>>> we can resolve them quickly.
> >>>>>
> >>>>> I agree that due to the big changes we should have a very thorough
> and
> >>>>> principled testing period where we put Flink through the paces.
> >>>>>
> >>>>> Cheers,
> >>>>> Till
> >>>>>
> >>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> [hidden email]>
> >>>>> wrote:
> >>>>>
> >>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> >>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
> >> local
> >>>>>> state recovery) are nearly done.
> >>>>>>
> >>>>>> I'm unsure about local state recovery, but I still see open issues
> for
> >>>>>> FLIP-6 and the network stack rework.
> >>>>>> As such it doesn't make sense to release 1.5 now.
> >>>>>>
> >>>>>> Given the large scope of these features I would very much prefer to
> >> have
> >>>>>> them active on master for a while before a feature-freeze
> >>>>>> to expose them to a wider audience.
> >>>>>>
> >>>>>> IMO it will take at least another month before we can start the
> >> release
> >>>>>> process for 1.5, i.e. the feature freeze.
> >>>>>> (2 more weeks for implementation, 2 weeks on master for the dust to
> >> settle)
> >>>>>>
> >>>>>>
> >>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> >>>>>>
> >>>>>>> Hi Aljoscha,
> >>>>>>>
> >>>>>>> I believe that support for Broadcast State should also be in 1.5.
> >>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
> >>>>>>> https://github.com/apache/flink/pull/5230> for that
> >>>>>>> and there are some pending issues related to scala api and
> >> documentation.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Kostas
> >>>>>>>
> >>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]>
> wrote:
> >>>>>>>> Hi Shuyi,
> >>>>>>>>
> >>>>>>>> I will take a look at it again this week. I'm pretty sure it will
> be
> >>>>>>>> part of 1.5.0.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Timo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> >>>>>>>>
> >>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot
> of
> >>>>>>>>> internal users waiting for this feature.
> >>>>>>>>>
> >>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
> >> Support
> >>>>>>>>> accessing subfields of a Composite element in an Object Array
> type
> >>>>>>>>> column
> >>>>>>>>>
> >>>>>>>>> Thanks a lot
> >>>>>>>>> Shuyi
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> [hidden email]
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi guys,
> >>>>>>>>>> Sorry for jumping in, but I think
> >>>>>>>>>>
> >>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> >>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible
> >> with
> >>>>>>>>>> Elasticsearch 5.2+ client
> >>>>>>>>>>
> >>>>>>>>>> have long been awaited and there was one PR from me and from
> >> someone
> >>>>>>>>>> else
> >>>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
> >> would
> >>>>>>>>>> be
> >>>>>>>>>> great!
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>> --
> >>>>>>>>>> Christophe
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> [hidden email]>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>> it would be great if we can include the first version of the
> SQL
> >>>>>>>>>>> client
> >>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> >> week. I
> >>>>>>>>>>> think
> >>>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
> >> is far
> >>>>>>>>>>>
> >>>>>>>>>> away
> >>>>>>>>>>
> >>>>>>>>>>> from feature completeness but will be a great tool for Flink
> >>>>>>>>>>> beginners.
> >>>>>>>>>>>
> >>>>>>>>>>> In order to use the SQL client we would need to also add some
> >> table
> >>>>>>>>>>> sources with the new unified table factories (FLINK-8535), but
> >> this is
> >>>>>>>>>>> optional because a user can implement own table factories at
> the
> >>>>>>>>>>>
> >>>>>>>>>> begining.
> >>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Timo
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks for starting the discussion.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think there’s a few connector related must-have improvements
> >> that
> >>>>>>>>>>>> we
> >>>>>>>>>>>> should get in before the feature freeze, since quite a few
> >> users have
> >>>>>>>>>>>>
> >>>>>>>>>>> been
> >>>>>>>>>>> asking for them:
> >>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> timestamp
> >> to
> >>>>>>>>>>>> set
> >>>>>>>>>>>>
> >>>>>>>>>>> up
> >>>>>>>>>>> start offset
> >>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> >> should
> >>>>>>>>>>>> consider idle partitions
> >>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >>>>>>>>>>>> FlinkKinesisConsumer
> >>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> >> FlinkKafkaConsumer
> >>>>>>>>>>>>
> >>>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479
> is
> >>>>>>>>>>>> still
> >>>>>>>>>>>> lacking a pull request.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Gordon
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> >>>>>>>>>>>>
> >>>>>>>>>>> [hidden email])
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>
> >>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> >> that to
> >>>>>>>>>>>> get
> >>>>>>>>>>>>
> >>>>>>>>>>> a
> >>>>>>>>>>> stable release out before putting in a couple of new features.
> >> Back
> >>>>>>>>>>> then,
> >>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> local
> >> state
> >>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> >> 1.5.0
> >>>>>>>>>>>> development cycle to allow for those features to become ready
> >> and
> >>>>>>>>>>>> then
> >>>>>>>>>>>>
> >>>>>>>>>>> do
> >>>>>>>>>>> the next release.
> >>>>>>>>>>>> We are now approaching the approximate time where we wanted to
> >> do the
> >>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
> >> gather
> >>>>>>>>>>>> opinions on how we should proceed now.
> >>>>>>>>>>>>
> >>>>>>>>>>>> With this, I'd also like to propose myself as the release
> >> manager for
> >>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> >>>>>>>>>>>> interested in
> >>>>>>>>>>>> doing that.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Aljoscha
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>> Christophe
> >>>>>>>>>>
> >>>>>>>>>>
> >>>
> >>
> >>
>
>

[Aljoscha Krettek-2]

End of last week was the day where we wanted to to the cut of the release branch. There are still a bunch of open blocker issues about bugs in our Jira: [1]. So I'm wondering whether we should actually cut the branch now because some commits would have to be merged on release-1.4, release-1.5, and master. What do you think?

Regarding managing the release: I will have two weeks in mid march where I won't have time and this could be the hot release phase. I think that because of this, it would be better to have a release manager other than me and I think Till would be a good candidate for that since he's the lead of FLIP-6 which is the prime new feature of the next release. What do you think about that?

[1] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20and%20priority%20%3D%20blocker%20and%20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved

> On 13. Feb 2018, at 10:00, Till Rohrmann <[hidden email]> wrote:
>
> +1 from my side.
>
> Cheers,
> Till
>
> On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <[hidden email]>
> wrote:
>
>> +0.95 from my side.
>>
>> Network changes are mostly reviewed and should be merged by the end of
>> this week.
>>
>> Piotrek
>>
>>> On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email]> wrote:
>>>
>>> I agree with the basic idea. I think there is no need to call it "soft
>>> feature freeze" though - it is a feature freeze (no new features get
>>> merged) ;-)
>>>
>>> What you are suggesting is to delay forking of the release-1.5 branch to
>>> avoid applying the bug fixes to too many branches. That makes sense.
>>> In effect, there is a period of one week (next week) where no post 1.5
>>> features can be merged (feature freeze but release branch not forked),
>>> which should be okay.
>>>
>>> Best,
>>> Stephan
>>>
>>>
>>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
>> [hidden email]
>>>> wrote:
>>>
>>>> For me as well +1.
>>>>
>>>> Cheers,
>>>> Kostas
>>>>
>>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]> wrote:
>>>>>
>>>>> Sounds good to me. +1 from my side.
>>>>>
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>>>>>> I agree with Chesnay: we should do a soft "feature freeze" first, were
>>>> we agree to not merge new features to master after that and then to the
>>>> actual hard cutting of the release branch a while later.
>>>>>>
>>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as soft
>>>> feature freeze and end of next week (23.02.2018) as the hard cut of the
>>>> release branch?
>>>>>>
>>>>>> What do you think?
>>>>>>
>>>>>> --
>>>>>> Aljoscha
>>>>>>
>>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]>
>> wrote:
>>>>>>>
>>>>>>> Local state recovery is almost completely done. Only some reviews and
>>>>>>> merging of the final PRs is pending.
>>>>>>>
>>>>>>> The network stack improvements are on a good way to be finished by
>> the
>>>> end
>>>>>>> of this week or beginning of next week. To my knowledge we got
>> recently
>>>>>>> green Travis builds :-) The network stack changes will also include
>> the
>>>>>>> application level flow control and the back pressure based checkpoint
>>>>>>> alignment. So only the last reviews and merging is missing.
>>>>>>>
>>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
>> default.
>>>>>>> There are still some smaller things left to be done but I'm confident
>>>> that
>>>>>>> we can resolve them quickly.
>>>>>>>
>>>>>>> I agree that due to the big changes we should have a very thorough
>> and
>>>>>>> principled testing period where we put Flink through the paces.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
>> [hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
>>>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
>>>> local
>>>>>>>> state recovery) are nearly done.
>>>>>>>>
>>>>>>>> I'm unsure about local state recovery, but I still see open issues
>> for
>>>>>>>> FLIP-6 and the network stack rework.
>>>>>>>> As such it doesn't make sense to release 1.5 now.
>>>>>>>>
>>>>>>>> Given the large scope of these features I would very much prefer to
>>>> have
>>>>>>>> them active on master for a while before a feature-freeze
>>>>>>>> to expose them to a wider audience.
>>>>>>>>
>>>>>>>> IMO it will take at least another month before we can start the
>>>> release
>>>>>>>> process for 1.5, i.e. the feature freeze.
>>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust to
>>>> settle)
>>>>>>>>
>>>>>>>>
>>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>>>>>>>>
>>>>>>>>> Hi Aljoscha,
>>>>>>>>>
>>>>>>>>> I believe that support for Broadcast State should also be in 1.5.
>>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
>>>>>>>>> https://github.com/apache/flink/pull/5230> for that
>>>>>>>>> and there are some pending issues related to scala api and
>>>> documentation.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Kostas
>>>>>>>>>
>>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]>
>> wrote:
>>>>>>>>>> Hi Shuyi,
>>>>>>>>>>
>>>>>>>>>> I will take a look at it again this week. I'm pretty sure it will
>> be
>>>>>>>>>> part of 1.5.0.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Timo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>>>>>>>>>>
>>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a lot
>> of
>>>>>>>>>>> internal users waiting for this feature.
>>>>>>>>>>>
>>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>]
>>>> Support
>>>>>>>>>>> accessing subfields of a Composite element in an Object Array
>> type
>>>>>>>>>>> column
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot
>>>>>>>>>>> Shuyi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
>> [hidden email]
>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>> Sorry for jumping in, but I think
>>>>>>>>>>>>
>>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>>>>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not compatible
>>>> with
>>>>>>>>>>>> Elasticsearch 5.2+ client
>>>>>>>>>>>>
>>>>>>>>>>>> have long been awaited and there was one PR from me and from
>>>> someone
>>>>>>>>>>>> else
>>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5 that
>>>> would
>>>>>>>>>>>> be
>>>>>>>>>>>> great!
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> --
>>>>>>>>>>>> Christophe
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
>> [hidden email]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>>>> it would be great if we can include the first version of the
>> SQL
>>>>>>>>>>>>> client
>>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
>>>> week. I
>>>>>>>>>>>>> think
>>>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status. It
>>>> is far
>>>>>>>>>>>>>
>>>>>>>>>>>> away
>>>>>>>>>>>>
>>>>>>>>>>>>> from feature completeness but will be a great tool for Flink
>>>>>>>>>>>>> beginners.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In order to use the SQL client we would need to also add some
>>>> table
>>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535), but
>>>> this is
>>>>>>>>>>>>> optional because a user can implement own table factories at
>> the
>>>>>>>>>>>>>
>>>>>>>>>>>> begining.
>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Timo
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Aljoscha,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for starting the discussion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think there’s a few connector related must-have improvements
>>>> that
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
>>>> users have
>>>>>>>>>>>>>>
>>>>>>>>>>>>> been
>>>>>>>>>>>>> asking for them:
>>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
>> timestamp
>>>> to
>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>
>>>>>>>>>>>>> up
>>>>>>>>>>>>> start offset
>>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
>>>> should
>>>>>>>>>>>>>> consider idle partitions
>>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>>>>>>>>>>>>>> FlinkKinesisConsumer
>>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
>>>> FlinkKafkaConsumer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These are still missing in the master branch. Only FLINK-5479
>> is
>>>>>>>>>>>>>> still
>>>>>>>>>>>>>> lacking a pull request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Gordon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>>>>>>>>>>>>>>
>>>>>>>>>>>>> [hidden email])
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
>>>> that to
>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>
>>>>>>>>>>>>> a
>>>>>>>>>>>>> stable release out before putting in a couple of new features.
>>>> Back
>>>>>>>>>>>>> then,
>>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
>> local
>>>> state
>>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
>>>> 1.5.0
>>>>>>>>>>>>>> development cycle to allow for those features to become ready
>>>> and
>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>
>>>>>>>>>>>>> do
>>>>>>>>>>>>> the next release.
>>>>>>>>>>>>>> We are now approaching the approximate time where we wanted to
>>>> do the
>>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are and
>>>> gather
>>>>>>>>>>>>>> opinions on how we should proceed now.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
>>>> manager for
>>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
>>>>>>>>>>>>>> interested in
>>>>>>>>>>>>>> doing that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>> Christophe
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>
>>>>
>>>>
>>
>>

[Till Rohrmann]

+1 for cutting the release branch now.

I'm also happy to volunteer as the release manager for Flink 1.5.

Cheers,
Till

On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <[hidden email]>
wrote:

> End of last week was the day where we wanted to to the cut of the release
> branch. There are still a bunch of open blocker issues about bugs in our
> Jira: [1]. So I'm wondering whether we should actually cut the branch now
> because some commits would have to be merged on release-1.4, release-1.5,
> and master. What do you think?
>
> Regarding managing the release: I will have two weeks in mid march where I
> won't have time and this could be the hot release phase. I think that
> because of this, it would be better to have a release manager other than me
> and I think Till would be a good candidate for that since he's the lead of
> FLIP-6 which is the prime new feature of the next release. What do you
> think about that?
>
> [1] <a href="https://issues.apache.org/jira/issues/?jql=project%20%">https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
> 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
>
> > On 13. Feb 2018, at 10:00, Till Rohrmann <[hidden email]> wrote:
> >
> > +1 from my side.
> >
> > Cheers,
> > Till
> >
> > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <[hidden email]
> >
> > wrote:
> >
> >> +0.95 from my side.
> >>
> >> Network changes are mostly reviewed and should be merged by the end of
> >> this week.
> >>
> >> Piotrek
> >>
> >>> On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email]> wrote:
> >>>
> >>> I agree with the basic idea. I think there is no need to call it "soft
> >>> feature freeze" though - it is a feature freeze (no new features get
> >>> merged) ;-)
> >>>
> >>> What you are suggesting is to delay forking of the release-1.5 branch
> to
> >>> avoid applying the bug fixes to too many branches. That makes sense.
> >>> In effect, there is a period of one week (next week) where no post 1.5
> >>> features can be merged (feature freeze but release branch not forked),
> >>> which should be okay.
> >>>
> >>> Best,
> >>> Stephan
> >>>
> >>>
> >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> >> [hidden email]
> >>>> wrote:
> >>>
> >>>> For me as well +1.
> >>>>
> >>>> Cheers,
> >>>> Kostas
> >>>>
> >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]>
> wrote:
> >>>>>
> >>>>> Sounds good to me. +1 from my side.
> >>>>>
> >>>>> Regards,
> >>>>> Timo
> >>>>>
> >>>>>
> >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
> were
> >>>> we agree to not merge new features to master after that and then to
> the
> >>>> actual hard cutting of the release branch a while later.
> >>>>>>
> >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
> soft
> >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
> the
> >>>> release branch?
> >>>>>>
> >>>>>> What do you think?
> >>>>>>
> >>>>>> --
> >>>>>> Aljoscha
> >>>>>>
> >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]>
> >> wrote:
> >>>>>>>
> >>>>>>> Local state recovery is almost completely done. Only some reviews
> and
> >>>>>>> merging of the final PRs is pending.
> >>>>>>>
> >>>>>>> The network stack improvements are on a good way to be finished by
> >> the
> >>>> end
> >>>>>>> of this week or beginning of next week. To my knowledge we got
> >> recently
> >>>>>>> green Travis builds :-) The network stack changes will also include
> >> the
> >>>>>>> application level flow control and the back pressure based
> checkpoint
> >>>>>>> alignment. So only the last reviews and merging is missing.
> >>>>>>>
> >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> >> default.
> >>>>>>> There are still some smaller things left to be done but I'm
> confident
> >>>> that
> >>>>>>> we can resolve them quickly.
> >>>>>>>
> >>>>>>> I agree that due to the big changes we should have a very thorough
> >> and
> >>>>>>> principled testing period where we put Flink through the paces.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Till
> >>>>>>>
> >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> >> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> >>>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
> >>>> local
> >>>>>>>> state recovery) are nearly done.
> >>>>>>>>
> >>>>>>>> I'm unsure about local state recovery, but I still see open issues
> >> for
> >>>>>>>> FLIP-6 and the network stack rework.
> >>>>>>>> As such it doesn't make sense to release 1.5 now.
> >>>>>>>>
> >>>>>>>> Given the large scope of these features I would very much prefer
> to
> >>>> have
> >>>>>>>> them active on master for a while before a feature-freeze
> >>>>>>>> to expose them to a wider audience.
> >>>>>>>>
> >>>>>>>> IMO it will take at least another month before we can start the
> >>>> release
> >>>>>>>> process for 1.5, i.e. the feature freeze.
> >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust
> to
> >>>> settle)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> >>>>>>>>
> >>>>>>>>> Hi Aljoscha,
> >>>>>>>>>
> >>>>>>>>> I believe that support for Broadcast State should also be in 1.5.
> >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <
> >>>>>>>>> https://github.com/apache/flink/pull/5230> for that
> >>>>>>>>> and there are some pending issues related to scala api and
> >>>> documentation.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Kostas
> >>>>>>>>>
> >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]>
> >> wrote:
> >>>>>>>>>> Hi Shuyi,
> >>>>>>>>>>
> >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
> will
> >> be
> >>>>>>>>>> part of 1.5.0.
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Timo
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
> lot
> >> of
> >>>>>>>>>>> internal users waiting for this feature.
> >>>>>>>>>>>
> >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923
> >]
> >>>> Support
> >>>>>>>>>>> accessing subfields of a Composite element in an Object Array
> >> type
> >>>>>>>>>>> column
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks a lot
> >>>>>>>>>>> Shuyi
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> >> [hidden email]
> >>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi guys,
> >>>>>>>>>>>> Sorry for jumping in, but I think
> >>>>>>>>>>>>
> >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> >>>>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not
> compatible
> >>>> with
> >>>>>>>>>>>> Elasticsearch 5.2+ client
> >>>>>>>>>>>>
> >>>>>>>>>>>> have long been awaited and there was one PR from me and from
> >>>> someone
> >>>>>>>>>>>> else
> >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
> that
> >>>> would
> >>>>>>>>>>>> be
> >>>>>>>>>>>> great!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Christophe
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> >> [hidden email]>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>>>> it would be great if we can include the first version of the
> >> SQL
> >>>>>>>>>>>>> client
> >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> >>>> week. I
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status.
> It
> >>>> is far
> >>>>>>>>>>>>>
> >>>>>>>>>>>> away
> >>>>>>>>>>>>
> >>>>>>>>>>>>> from feature completeness but will be a great tool for Flink
> >>>>>>>>>>>>> beginners.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In order to use the SQL client we would need to also add some
> >>>> table
> >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
> but
> >>>> this is
> >>>>>>>>>>>>> optional because a user can implement own table factories at
> >> the
> >>>>>>>>>>>>>
> >>>>>>>>>>>> begining.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Timo
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Aljoscha,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for starting the discussion.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think there’s a few connector related must-have
> improvements
> >>>> that
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
> >>>> users have
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> been
> >>>>>>>>>>>>> asking for them:
> >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> >> timestamp
> >>>> to
> >>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> up
> >>>>>>>>>>>>> start offset
> >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> >>>> should
> >>>>>>>>>>>>>> consider idle partitions
> >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> >>>>>>>>>>>>>> FlinkKinesisConsumer
> >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> >>>> FlinkKafkaConsumer
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> These are still missing in the master branch. Only
> FLINK-5479
> >> is
> >>>>>>>>>>>>>> still
> >>>>>>>>>>>>>> lacking a pull request.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>> Gordon
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> [hidden email])
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> Hi Everyone,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> >>>> that to
> >>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>> stable release out before putting in a couple of new
> features.
> >>>> Back
> >>>>>>>>>>>>> then,
> >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> >> local
> >>>> state
> >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> >>>> 1.5.0
> >>>>>>>>>>>>>> development cycle to allow for those features to become
> ready
> >>>> and
> >>>>>>>>>>>>>> then
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> do
> >>>>>>>>>>>>> the next release.
> >>>>>>>>>>>>>> We are now approaching the approximate time where we wanted
> to
> >>>> do the
> >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
> and
> >>>> gather
> >>>>>>>>>>>>>> opinions on how we should proceed now.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
> >>>> manager for
> >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> >>>>>>>>>>>>>> interested in
> >>>>>>>>>>>>>> doing that.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Aljoscha
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>> Christophe
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>

[bowen.li]

Hi guys,

Can we get FLINK-8667 <https://github.com/apache/flink/pull/5500> into
1.5.0? I've been waiting for it for quite a few days.

The reason I really want to get it into 1.5.0 is because KeyedBroadcastProce
ssFunction will be released in 1.5.0 and there's no compatibility issues.
If we don't get FLINK-8667 in, it will be much harder to make it work in
post-1.5.0 branches and migrate user apps.

Thanks,
Bowen



On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann <[hidden email]> wrote:

> +1 for cutting the release branch now.
>
> I'm also happy to volunteer as the release manager for Flink 1.5.
>
> Cheers,
> Till
>
> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > End of last week was the day where we wanted to to the cut of the release
> > branch. There are still a bunch of open blocker issues about bugs in our
> > Jira: [1]. So I'm wondering whether we should actually cut the branch now
> > because some commits would have to be merged on release-1.4, release-1.5,
> > and master. What do you think?
> >
> > Regarding managing the release: I will have two weeks in mid march where
> I
> > won't have time and this could be the hot release phase. I think that
> > because of this, it would be better to have a release manager other than
> me
> > and I think Till would be a good candidate for that since he's the lead
> of
> > FLIP-6 which is the prime new feature of the next release. What do you
> > think about that?
> >
> > [1] <a href="https://issues.apache.org/jira/issues/?jql=project%20%">https://issues.apache.org/jira/issues/?jql=project%20%
> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
> >
> > > On 13. Feb 2018, at 10:00, Till Rohrmann <[hidden email]> wrote:
> > >
> > > +1 from my side.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <
> [hidden email]
> > >
> > > wrote:
> > >
> > >> +0.95 from my side.
> > >>
> > >> Network changes are mostly reviewed and should be merged by the end of
> > >> this week.
> > >>
> > >> Piotrek
> > >>
> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email]> wrote:
> > >>>
> > >>> I agree with the basic idea. I think there is no need to call it
> "soft
> > >>> feature freeze" though - it is a feature freeze (no new features get
> > >>> merged) ;-)
> > >>>
> > >>> What you are suggesting is to delay forking of the release-1.5 branch
> > to
> > >>> avoid applying the bug fixes to too many branches. That makes sense.
> > >>> In effect, there is a period of one week (next week) where no post
> 1.5
> > >>> features can be merged (feature freeze but release branch not
> forked),
> > >>> which should be okay.
> > >>>
> > >>> Best,
> > >>> Stephan
> > >>>
> > >>>
> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> > >> [hidden email]
> > >>>> wrote:
> > >>>
> > >>>> For me as well +1.
> > >>>>
> > >>>> Cheers,
> > >>>> Kostas
> > >>>>
> > >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]>
> > wrote:
> > >>>>>
> > >>>>> Sounds good to me. +1 from my side.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Timo
> > >>>>>
> > >>>>>
> > >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> > >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
> > were
> > >>>> we agree to not merge new features to master after that and then to
> > the
> > >>>> actual hard cutting of the release branch a while later.
> > >>>>>>
> > >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
> > soft
> > >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
> > the
> > >>>> release branch?
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Aljoscha
> > >>>>>>
> > >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> Local state recovery is almost completely done. Only some reviews
> > and
> > >>>>>>> merging of the final PRs is pending.
> > >>>>>>>
> > >>>>>>> The network stack improvements are on a good way to be finished
> by
> > >> the
> > >>>> end
> > >>>>>>> of this week or beginning of next week. To my knowledge we got
> > >> recently
> > >>>>>>> green Travis builds :-) The network stack changes will also
> include
> > >> the
> > >>>>>>> application level flow control and the back pressure based
> > checkpoint
> > >>>>>>> alignment. So only the last reviews and merging is missing.
> > >>>>>>>
> > >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> > >> default.
> > >>>>>>> There are still some smaller things left to be done but I'm
> > confident
> > >>>> that
> > >>>>>>> we can resolve them quickly.
> > >>>>>>>
> > >>>>>>> I agree that due to the big changes we should have a very
> thorough
> > >> and
> > >>>>>>> principled testing period where we put Flink through the paces.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> > >> [hidden email]>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> > >>>>>>>> assumption that the 3 big features (FLIP-6, network stack
> changes,
> > >>>> local
> > >>>>>>>> state recovery) are nearly done.
> > >>>>>>>>
> > >>>>>>>> I'm unsure about local state recovery, but I still see open
> issues
> > >> for
> > >>>>>>>> FLIP-6 and the network stack rework.
> > >>>>>>>> As such it doesn't make sense to release 1.5 now.
> > >>>>>>>>
> > >>>>>>>> Given the large scope of these features I would very much prefer
> > to
> > >>>> have
> > >>>>>>>> them active on master for a while before a feature-freeze
> > >>>>>>>> to expose them to a wider audience.
> > >>>>>>>>
> > >>>>>>>> IMO it will take at least another month before we can start the
> > >>>> release
> > >>>>>>>> process for 1.5, i.e. the feature freeze.
> > >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust
> > to
> > >>>> settle)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>
> > >>>>>>>>> I believe that support for Broadcast State should also be in
> 1.5.
> > >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230
> <
> > >>>>>>>>> https://github.com/apache/flink/pull/5230> for that
> > >>>>>>>>> and there are some pending issues related to scala api and
> > >>>> documentation.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Kostas
> > >>>>>>>>>
> > >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]>
> > >> wrote:
> > >>>>>>>>>> Hi Shuyi,
> > >>>>>>>>>>
> > >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
> > will
> > >> be
> > >>>>>>>>>> part of 1.5.0.
> > >>>>>>>>>>
> > >>>>>>>>>> Regards,
> > >>>>>>>>>> Timo
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
> > lot
> > >> of
> > >>>>>>>>>>> internal users waiting for this feature.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/
> jira/browse/FLINK-7923
> > >]
> > >>>> Support
> > >>>>>>>>>>> accessing subfields of a Composite element in an Object Array
> > >> type
> > >>>>>>>>>>> column
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks a lot
> > >>>>>>>>>>> Shuyi
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> > >> [hidden email]
> > >>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi guys,
> > >>>>>>>>>>>> Sorry for jumping in, but I think
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> > >>>>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not
> > compatible
> > >>>> with
> > >>>>>>>>>>>> Elasticsearch 5.2+ client
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> have long been awaited and there was one PR from me and from
> > >>>> someone
> > >>>>>>>>>>>> else
> > >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
> > that
> > >>>> would
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>> great!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> > >> [hidden email]>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>> it would be great if we can include the first version of
> the
> > >> SQL
> > >>>>>>>>>>>>> client
> > >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> > >>>> week. I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha"
> status.
> > It
> > >>>> is far
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> away
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> from feature completeness but will be a great tool for
> Flink
> > >>>>>>>>>>>>> beginners.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In order to use the SQL client we would need to also add
> some
> > >>>> table
> > >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
> > but
> > >>>> this is
> > >>>>>>>>>>>>> optional because a user can implement own table factories
> at
> > >> the
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> begining.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>> Timo
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for starting the discussion.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think there’s a few connector related must-have
> > improvements
> > >>>> that
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
> > >>>> users have
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> been
> > >>>>>>>>>>>>> asking for them:
> > >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> > >> timestamp
> > >>>> to
> > >>>>>>>>>>>>>> set
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> up
> > >>>>>>>>>>>>> start offset
> > >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in
> FlinkKafkaConsumer
> > >>>> should
> > >>>>>>>>>>>>>> consider idle partitions
> > >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> > >>>>>>>>>>>>>> FlinkKinesisConsumer
> > >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> > >>>> FlinkKafkaConsumer
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> These are still missing in the master branch. Only
> > FLINK-5479
> > >> is
> > >>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>> lacking a pull request.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>> Gordon
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> [hidden email])
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>> Hi Everyone,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we
> did
> > >>>> that to
> > >>>>>>>>>>>>>> get
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>> stable release out before putting in a couple of new
> > features.
> > >>>> Back
> > >>>>>>>>>>>>> then,
> > >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> > >> local
> > >>>> state
> > >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a
> shortened
> > >>>> 1.5.0
> > >>>>>>>>>>>>>> development cycle to allow for those features to become
> > ready
> > >>>> and
> > >>>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> do
> > >>>>>>>>>>>>> the next release.
> > >>>>>>>>>>>>>> We are now approaching the approximate time where we
> wanted
> > to
> > >>>> do the
> > >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
> > and
> > >>>> gather
> > >>>>>>>>>>>>>> opinions on how we should proceed now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
> > >>>> manager for
> > >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> > >>>>>>>>>>>>>> interested in
> > >>>>>>>>>>>>>> doing that.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> What do you think?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Aljoscha
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

[Till Rohrmann]

The release branch is now created [1]. Please be aware that we should only
commit bug fixes to this branch henceforth.

@Bowen, let's wait what Aljoscha says concerning FLINK-8667. If he agrees,
then we can still merge it into the release branch.

[1]
https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5

Cheers,
Till

On Mon, Feb 26, 2018 at 8:03 PM, Bowen Li <[hidden email]> wrote:

> Hi guys,
>
> Can we get FLINK-8667 <https://github.com/apache/flink/pull/5500> into
> 1.5.0? I've been waiting for it for quite a few days.
>
> The reason I really want to get it into 1.5.0 is because
> KeyedBroadcastProcessFunction will be released in 1.5.0 and there's no
> compatibility issues. If we don't get FLINK-8667 in, it will be much harder
> to make it work in post-1.5.0 branches and migrate user apps.
>
> Thanks,
> Bowen
>
>
>
> On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann <[hidden email]>
> wrote:
>
>> +1 for cutting the release branch now.
>>
>> I'm also happy to volunteer as the release manager for Flink 1.5.
>>
>> Cheers,
>> Till
>>
>> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <[hidden email]>
>> wrote:
>>
>> > End of last week was the day where we wanted to to the cut of the
>> release
>> > branch. There are still a bunch of open blocker issues about bugs in our
>> > Jira: [1]. So I'm wondering whether we should actually cut the branch
>> now
>> > because some commits would have to be merged on release-1.4,
>> release-1.5,
>> > and master. What do you think?
>> >
>> > Regarding managing the release: I will have two weeks in mid march
>> where I
>> > won't have time and this could be the hot release phase. I think that
>> > because of this, it would be better to have a release manager other
>> than me
>> > and I think Till would be a good candidate for that since he's the lead
>> of
>> > FLIP-6 which is the prime new feature of the next release. What do you
>> > think about that?
>> >
>> > [1] <a href="https://issues.apache.org/jira/issues/?jql=project%20%">https://issues.apache.org/jira/issues/?jql=project%20%
>> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
>> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
>> >
>> > > On 13. Feb 2018, at 10:00, Till Rohrmann <[hidden email]>
>> wrote:
>> > >
>> > > +1 from my side.
>> > >
>> > > Cheers,
>> > > Till
>> > >
>> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <
>> [hidden email]
>> > >
>> > > wrote:
>> > >
>> > >> +0.95 from my side.
>> > >>
>> > >> Network changes are mostly reviewed and should be merged by the end
>> of
>> > >> this week.
>> > >>
>> > >> Piotrek
>> > >>
>> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email]> wrote:
>> > >>>
>> > >>> I agree with the basic idea. I think there is no need to call it
>> "soft
>> > >>> feature freeze" though - it is a feature freeze (no new features get
>> > >>> merged) ;-)
>> > >>>
>> > >>> What you are suggesting is to delay forking of the release-1.5
>> branch
>> > to
>> > >>> avoid applying the bug fixes to too many branches. That makes sense.
>> > >>> In effect, there is a period of one week (next week) where no post
>> 1.5
>> > >>> features can be merged (feature freeze but release branch not
>> forked),
>> > >>> which should be okay.
>> > >>>
>> > >>> Best,
>> > >>> Stephan
>> > >>>
>> > >>>
>> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
>> > >> [hidden email]
>> > >>>> wrote:
>> > >>>
>> > >>>> For me as well +1.
>> > >>>>
>> > >>>> Cheers,
>> > >>>> Kostas
>> > >>>>
>> > >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email]>
>> > wrote:
>> > >>>>>
>> > >>>>> Sounds good to me. +1 from my side.
>> > >>>>>
>> > >>>>> Regards,
>> > >>>>> Timo
>> > >>>>>
>> > >>>>>
>> > >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
>> > >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
>> > were
>> > >>>> we agree to not merge new features to master after that and then to
>> > the
>> > >>>> actual hard cutting of the release branch a while later.
>> > >>>>>>
>> > >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
>> > soft
>> > >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
>> > the
>> > >>>> release branch?
>> > >>>>>>
>> > >>>>>> What do you think?
>> > >>>>>>
>> > >>>>>> --
>> > >>>>>> Aljoscha
>> > >>>>>>
>> > >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email]>
>> > >> wrote:
>> > >>>>>>>
>> > >>>>>>> Local state recovery is almost completely done. Only some
>> reviews
>> > and
>> > >>>>>>> merging of the final PRs is pending.
>> > >>>>>>>
>> > >>>>>>> The network stack improvements are on a good way to be finished
>> by
>> > >> the
>> > >>>> end
>> > >>>>>>> of this week or beginning of next week. To my knowledge we got
>> > >> recently
>> > >>>>>>> green Travis builds :-) The network stack changes will also
>> include
>> > >> the
>> > >>>>>>> application level flow control and the back pressure based
>> > checkpoint
>> > >>>>>>> alignment. So only the last reviews and merging is missing.
>> > >>>>>>>
>> > >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
>> > >> default.
>> > >>>>>>> There are still some smaller things left to be done but I'm
>> > confident
>> > >>>> that
>> > >>>>>>> we can resolve them quickly.
>> > >>>>>>>
>> > >>>>>>> I agree that due to the big changes we should have a very
>> thorough
>> > >> and
>> > >>>>>>> principled testing period where we put Flink through the paces.
>> > >>>>>>>
>> > >>>>>>> Cheers,
>> > >>>>>>> Till
>> > >>>>>>>
>> > >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
>> > >> [hidden email]>
>> > >>>>>>> wrote:
>> > >>>>>>>
>> > >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on
>> the
>> > >>>>>>>> assumption that the 3 big features (FLIP-6, network stack
>> changes,
>> > >>>> local
>> > >>>>>>>> state recovery) are nearly done.
>> > >>>>>>>>
>> > >>>>>>>> I'm unsure about local state recovery, but I still see open
>> issues
>> > >> for
>> > >>>>>>>> FLIP-6 and the network stack rework.
>> > >>>>>>>> As such it doesn't make sense to release 1.5 now.
>> > >>>>>>>>
>> > >>>>>>>> Given the large scope of these features I would very much
>> prefer
>> > to
>> > >>>> have
>> > >>>>>>>> them active on master for a while before a feature-freeze
>> > >>>>>>>> to expose them to a wider audience.
>> > >>>>>>>>
>> > >>>>>>>> IMO it will take at least another month before we can start the
>> > >>>> release
>> > >>>>>>>> process for 1.5, i.e. the feature freeze.
>> > >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the
>> dust
>> > to
>> > >>>> settle)
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi Aljoscha,
>> > >>>>>>>>>
>> > >>>>>>>>> I believe that support for Broadcast State should also be in
>> 1.5.
>> > >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230
>> <
>> > >>>>>>>>> https://github.com/apache/flink/pull/5230> for that
>> > >>>>>>>>> and there are some pending issues related to scala api and
>> > >>>> documentation.
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks,
>> > >>>>>>>>> Kostas
>> > >>>>>>>>>
>> > >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email]>
>> > >> wrote:
>> > >>>>>>>>>> Hi Shuyi,
>> > >>>>>>>>>>
>> > >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
>> > will
>> > >> be
>> > >>>>>>>>>> part of 1.5.0.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Regards,
>> > >>>>>>>>>> Timo
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
>> > lot
>> > >> of
>> > >>>>>>>>>>> internal users waiting for this feature.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jir
>> a/browse/FLINK-7923
>> > >]
>> > >>>> Support
>> > >>>>>>>>>>> accessing subfields of a Composite element in an Object
>> Array
>> > >> type
>> > >>>>>>>>>>> column
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Thanks a lot
>> > >>>>>>>>>>> Shuyi
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
>> > >> [hidden email]
>> > >>>>>
>> > >>>>>>>>>>> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Hi guys,
>> > >>>>>>>>>>>> Sorry for jumping in, but I think
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
>> > >>>>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not
>> > compatible
>> > >>>> with
>> > >>>>>>>>>>>> Elasticsearch 5.2+ client
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> have long been awaited and there was one PR from me and
>> from
>> > >>>> someone
>> > >>>>>>>>>>>> else
>> > >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
>> > that
>> > >>>> would
>> > >>>>>>>>>>>> be
>> > >>>>>>>>>>>> great!
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Thanks!
>> > >>>>>>>>>>>> --
>> > >>>>>>>>>>>> Christophe
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
>> > >> [hidden email]>
>> > >>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Hi Aljoscha,
>> > >>>>>>>>>>>>> it would be great if we can include the first version of
>> the
>> > >> SQL
>> > >>>>>>>>>>>>> client
>> > >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR
>> this
>> > >>>> week. I
>> > >>>>>>>>>>>>> think
>> > >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha"
>> status.
>> > It
>> > >>>> is far
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>> away
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> from feature completeness but will be a great tool for
>> Flink
>> > >>>>>>>>>>>>> beginners.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> In order to use the SQL client we would need to also add
>> some
>> > >>>> table
>> > >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
>> > but
>> > >>>> this is
>> > >>>>>>>>>>>>> optional because a user can implement own table factories
>> at
>> > >> the
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>> begining.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> Regards,
>> > >>>>>>>>>>>>> Timo
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Hi Aljoscha,
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Thanks for starting the discussion.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> I think there’s a few connector related must-have
>> > improvements
>> > >>>> that
>> > >>>>>>>>>>>>>> we
>> > >>>>>>>>>>>>>> should get in before the feature freeze, since quite a
>> few
>> > >>>> users have
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> been
>> > >>>>>>>>>>>>> asking for them:
>> > >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
>> > >> timestamp
>> > >>>> to
>> > >>>>>>>>>>>>>> set
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> up
>> > >>>>>>>>>>>>> start offset
>> > >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in
>> FlinkKafkaConsumer
>> > >>>> should
>> > >>>>>>>>>>>>>> consider idle partitions
>> > >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
>> > >>>>>>>>>>>>>> FlinkKinesisConsumer
>> > >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
>> > >>>> FlinkKafkaConsumer
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> These are still missing in the master branch. Only
>> > FLINK-5479
>> > >> is
>> > >>>>>>>>>>>>>> still
>> > >>>>>>>>>>>>>> lacking a pull request.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Cheers,
>> > >>>>>>>>>>>>>> Gordon
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> [hidden email])
>> > >>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>> Hi Everyone,
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we
>> did
>> > >>>> that to
>> > >>>>>>>>>>>>>> get
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> a
>> > >>>>>>>>>>>>> stable release out before putting in a couple of new
>> > features.
>> > >>>> Back
>> > >>>>>>>>>>>>> then,
>> > >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
>> > >> local
>> > >>>> state
>> > >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a
>> shortened
>> > >>>> 1.5.0
>> > >>>>>>>>>>>>>> development cycle to allow for those features to become
>> > ready
>> > >>>> and
>> > >>>>>>>>>>>>>> then
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> do
>> > >>>>>>>>>>>>> the next release.
>> > >>>>>>>>>>>>>> We are now approaching the approximate time where we
>> wanted
>> > to
>> > >>>> do the
>> > >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
>> > and
>> > >>>> gather
>> > >>>>>>>>>>>>>> opinions on how we should proceed now.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
>> > >>>> manager for
>> > >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would
>> be
>> > >>>>>>>>>>>>>> interested in
>> > >>>>>>>>>>>>>> doing that.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> What do you think?
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>>> Aljoscha
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>> --
>> > >>>>>>>>>>>> Christophe
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>
>> > >>>>
>> > >>>>
>> > >>
>> > >>
>> >
>> >
>>
>
>

[Aljoscha Krettek-2]

@Bowen Yes, we're currently looking at getting this in because, as you said, future changes would break the API.

> On 27. Feb 2018, at 15:15, Till Rohrmann <[hidden email]> wrote:
>
> The release branch is now created [1]. Please be aware that we should only commit bug fixes to this branch henceforth.
>
> @Bowen, let's wait what Aljoscha says concerning FLINK-8667. If he agrees, then we can still merge it into the release branch.
>
> [1] https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5 <https://git-wip-us.apache.org/repos/asf?p=flink.git;a=shortlog;h=refs/heads/release-1.5>
>
> Cheers,
> Till
>
> On Mon, Feb 26, 2018 at 8:03 PM, Bowen Li <[hidden email] <mailto:[hidden email]>> wrote:
> Hi guys,
>
> Can we get FLINK-8667 <https://github.com/apache/flink/pull/5500> into 1.5.0? I've been waiting for it for quite a few days.
>
> The reason I really want to get it into 1.5.0 is because KeyedBroadcastProcessFunction will be released in 1.5.0 and there's no compatibility issues. If we don't get FLINK-8667 in, it will be much harder to make it work in post-1.5.0 branches and migrate user apps.
>
> Thanks,
> Bowen
>
>
>
> On Mon, Feb 26, 2018 at 7:21 AM, Till Rohrmann <[hidden email] <mailto:[hidden email]>> wrote:
> +1 for cutting the release branch now.
>
> I'm also happy to volunteer as the release manager for Flink 1.5.
>
> Cheers,
> Till
>
> On Mon, Feb 26, 2018 at 3:45 PM, Aljoscha Krettek <[hidden email] <mailto:[hidden email]>>
> wrote:
>
> > End of last week was the day where we wanted to to the cut of the release
> > branch. There are still a bunch of open blocker issues about bugs in our
> > Jira: [1]. So I'm wondering whether we should actually cut the branch now
> > because some commits would have to be merged on release-1.4, release-1.5,
> > and master. What do you think?
> >
> > Regarding managing the release: I will have two weeks in mid march where I
> > won't have time and this could be the hot release phase. I think that
> > because of this, it would be better to have a release manager other than me
> > and I think Till would be a good candidate for that since he's the lead of
> > FLIP-6 which is the prime new feature of the next release. What do you
> > think about that?
> >
> > [1] <a href="https://issues.apache.org/jira/issues/?jql=project%20%">https://issues.apache.org/jira/issues/?jql=project%20% <<a href="https://issues.apache.org/jira/issues/?jql=project%20%">https://issues.apache.org/jira/issues/?jql=project%20%>
> > 3D%20FLINK%20and%20priority%20%3D%20blocker%20and%
> > 20fixVersion%20%3D%201.5.0%20and%20resolution%20%3D%20unresolved
> >
> > > On 13. Feb 2018, at 10:00, Till Rohrmann <[hidden email] <mailto:[hidden email]>> wrote:
> > >
> > > +1 from my side.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Tue, Feb 13, 2018 at 9:52 AM, Piotr Nowojski <[hidden email] <mailto:[hidden email]>
> > >
> > > wrote:
> > >
> > >> +0.95 from my side.
> > >>
> > >> Network changes are mostly reviewed and should be merged by the end of
> > >> this week.
> > >>
> > >> Piotrek
> > >>
> > >>> On 12 Feb 2018, at 17:41, Stephan Ewen <[hidden email] <mailto:[hidden email]>> wrote:
> > >>>
> > >>> I agree with the basic idea. I think there is no need to call it "soft
> > >>> feature freeze" though - it is a feature freeze (no new features get
> > >>> merged) ;-)
> > >>>
> > >>> What you are suggesting is to delay forking of the release-1.5 branch
> > to
> > >>> avoid applying the bug fixes to too many branches. That makes sense.
> > >>> In effect, there is a period of one week (next week) where no post 1.5
> > >>> features can be merged (feature freeze but release branch not forked),
> > >>> which should be okay.
> > >>>
> > >>> Best,
> > >>> Stephan
> > >>>
> > >>>
> > >>> On Mon, Feb 12, 2018 at 3:00 PM, Kostas Kloudas <
> > >> [hidden email] <mailto:[hidden email]>
> > >>>> wrote:
> > >>>
> > >>>> For me as well +1.
> > >>>>
> > >>>> Cheers,
> > >>>> Kostas
> > >>>>
> > >>>>> On Feb 12, 2018, at 2:59 PM, Timo Walther <[hidden email] <mailto:[hidden email]>>
> > wrote:
> > >>>>>
> > >>>>> Sounds good to me. +1 from my side.
> > >>>>>
> > >>>>> Regards,
> > >>>>> Timo
> > >>>>>
> > >>>>>
> > >>>>> Am 2/12/18 um 2:56 PM schrieb Aljoscha Krettek:
> > >>>>>> I agree with Chesnay: we should do a soft "feature freeze" first,
> > were
> > >>>> we agree to not merge new features to master after that and then to
> > the
> > >>>> actual hard cutting of the release branch a while later.
> > >>>>>>
> > >>>>>> For actual dates, I'm proposing end of this week (16.02.2018) as
> > soft
> > >>>> feature freeze and end of next week (23.02.2018) as the hard cut of
> > the
> > >>>> release branch?
> > >>>>>>
> > >>>>>> What do you think?
> > >>>>>>
> > >>>>>> --
> > >>>>>> Aljoscha
> > >>>>>>
> > >>>>>>> On 8. Feb 2018, at 10:15, Till Rohrmann <[hidden email] <mailto:[hidden email]>>
> > >> wrote:
> > >>>>>>>
> > >>>>>>> Local state recovery is almost completely done. Only some reviews
> > and
> > >>>>>>> merging of the final PRs is pending.
> > >>>>>>>
> > >>>>>>> The network stack improvements are on a good way to be finished by
> > >> the
> > >>>> end
> > >>>>>>> of this week or beginning of next week. To my knowledge we got
> > >> recently
> > >>>>>>> green Travis builds :-) The network stack changes will also include
> > >> the
> > >>>>>>> application level flow control and the back pressure based
> > checkpoint
> > >>>>>>> alignment. So only the last reviews and merging is missing.
> > >>>>>>>
> > >>>>>>> Concerning Flip-6, I'm currently working on enabling Flip-6 by
> > >> default.
> > >>>>>>> There are still some smaller things left to be done but I'm
> > confident
> > >>>> that
> > >>>>>>> we can resolve them quickly.
> > >>>>>>>
> > >>>>>>> I agree that due to the big changes we should have a very thorough
> > >> and
> > >>>>>>> principled testing period where we put Flink through the paces.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Wed, Feb 7, 2018 at 10:55 AM, Chesnay Schepler <
> > >> [hidden email] <mailto:[hidden email]>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> As Aljoscha said we wanted to do 1.5 soon after 1.4 based on the
> > >>>>>>>> assumption that the 3 big features (FLIP-6, network stack changes,
> > >>>> local
> > >>>>>>>> state recovery) are nearly done.
> > >>>>>>>>
> > >>>>>>>> I'm unsure about local state recovery, but I still see open issues
> > >> for
> > >>>>>>>> FLIP-6 and the network stack rework.
> > >>>>>>>> As such it doesn't make sense to release 1.5 now.
> > >>>>>>>>
> > >>>>>>>> Given the large scope of these features I would very much prefer
> > to
> > >>>> have
> > >>>>>>>> them active on master for a while before a feature-freeze
> > >>>>>>>> to expose them to a wider audience.
> > >>>>>>>>
> > >>>>>>>> IMO it will take at least another month before we can start the
> > >>>> release
> > >>>>>>>> process for 1.5, i.e. the feature freeze.
> > >>>>>>>> (2 more weeks for implementation, 2 weeks on master for the dust
> > to
> > >>>> settle)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 05.02.2018 22:39, Kostas Kloudas wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>
> > >>>>>>>>> I believe that support for Broadcast State should also be in 1.5.
> > >>>>>>>>> There is an open PR https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230> <
> > >>>>>>>>> https://github.com/apache/flink/pull/5230 <https://github.com/apache/flink/pull/5230>> for that
> > >>>>>>>>> and there are some pending issues related to scala api and
> > >>>> documentation.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Kostas
> > >>>>>>>>>
> > >>>>>>>>> On Feb 5, 2018, at 5:37 PM, Timo Walther <[hidden email] <mailto:[hidden email]>>
> > >> wrote:
> > >>>>>>>>>> Hi Shuyi,
> > >>>>>>>>>>
> > >>>>>>>>>> I will take a look at it again this week. I'm pretty sure it
> > will
> > >> be
> > >>>>>>>>>> part of 1.5.0.
> > >>>>>>>>>>
> > >>>>>>>>>> Regards,
> > >>>>>>>>>> Timo
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Am 2/5/18 um 5:25 PM schrieb Shuyi Chen:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Aljoscha, can we get this feature in for 1.5.0? We have a
> > lot
> > >> of
> > >>>>>>>>>>> internal users waiting for this feature.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923 <https://issues.apache.org/jira/browse/FLINK-7923>
> > >]
> > >>>> Support
> > >>>>>>>>>>> accessing subfields of a Composite element in an Object Array
> > >> type
> > >>>>>>>>>>> column
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks a lot
> > >>>>>>>>>>> Shuyi
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Feb 5, 2018 at 6:59 AM, Christophe Jolif <
> > >> [hidden email] <mailto:[hidden email]>
> > >>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi guys,
> > >>>>>>>>>>>> Sorry for jumping in, but I think
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> [FLINK-8101] Elasticsearch 6.X support
> > >>>>>>>>>>>> [FLINK-7386]  Flink Elasticsearch 5 connector is not
> > compatible
> > >>>> with
> > >>>>>>>>>>>> Elasticsearch 5.2+ client
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> have long been awaited and there was one PR from me and from
> > >>>> someone
> > >>>>>>>>>>>> else
> > >>>>>>>>>>>> showing the interest ;) So if you could consider it for 1.5
> > that
> > >>>> would
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>> great!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, Feb 5, 2018 at 2:47 PM, Timo Walther <
> > >> [hidden email] <mailto:[hidden email]>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>> it would be great if we can include the first version of the
> > >> SQL
> > >>>>>>>>>>>>> client
> > >>>>>>>>>>>>> (see FLIP-24, Implementation Plan 1). I will open a PR this
> > >>>> week. I
> > >>>>>>>>>>>>> think
> > >>>>>>>>>>>>> we can merge this with explicit "experimental/alpha" status.
> > It
> > >>>> is far
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> away
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> from feature completeness but will be a great tool for Flink
> > >>>>>>>>>>>>> beginners.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> In order to use the SQL client we would need to also add some
> > >>>> table
> > >>>>>>>>>>>>> sources with the new unified table factories (FLINK-8535),
> > but
> > >>>> this is
> > >>>>>>>>>>>>> optional because a user can implement own table factories at
> > >> the
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> begining.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>> Timo
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Am 2/5/18 um 2:36 PM schrieb Tzu-Li (Gordon) Tai:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Aljoscha,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for starting the discussion.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I think there’s a few connector related must-have
> > improvements
> > >>>> that
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>> should get in before the feature freeze, since quite a few
> > >>>> users have
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> been
> > >>>>>>>>>>>>> asking for them:
> > >>>>>>>>>>>>>> [FLINK-6352] FlinkKafkaConsumer should support to use
> > >> timestamp
> > >>>> to
> > >>>>>>>>>>>>>> set
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> up
> > >>>>>>>>>>>>> start offset
> > >>>>>>>>>>>>>> [FLINK-5479] Per-partition watermarks in FlinkKafkaConsumer
> > >>>> should
> > >>>>>>>>>>>>>> consider idle partitions
> > >>>>>>>>>>>>>> [FLINK-8516] Pluggable shard-to-subtask partitioning for
> > >>>>>>>>>>>>>> FlinkKinesisConsumer
> > >>>>>>>>>>>>>> [FLINK-6109] Add a “checkpointed offset” metric to
> > >>>> FlinkKafkaConsumer
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> These are still missing in the master branch. Only
> > FLINK-5479
> > >> is
> > >>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>> lacking a pull request.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>> Gordon
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 31 January 2018 at 10:38:43 AM, Aljoscha Krettek (
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> [hidden email] <mailto:[hidden email]>)
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>> Hi Everyone,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> When we decided to do the 1.4.0 release a while back we did
> > >>>> that to
> > >>>>>>>>>>>>>> get
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>> stable release out before putting in a couple of new
> > features.
> > >>>> Back
> > >>>>>>>>>>>>> then,
> > >>>>>>>>>>>>> some of those new features (FLIP-6, network stack changes,
> > >> local
> > >>>> state
> > >>>>>>>>>>>>>> recovery) were almost ready and we wanted to do a shortened
> > >>>> 1.5.0
> > >>>>>>>>>>>>>> development cycle to allow for those features to become
> > ready
> > >>>> and
> > >>>>>>>>>>>>>> then
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> do
> > >>>>>>>>>>>>> the next release.
> > >>>>>>>>>>>>>> We are now approaching the approximate time where we wanted
> > to
> > >>>> do the
> > >>>>>>>>>>>>>> Flink 1.5.0 release so I would like to gauge where we are
> > and
> > >>>> gather
> > >>>>>>>>>>>>>> opinions on how we should proceed now.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> With this, I'd also like to propose myself as the release
> > >>>> manager for
> > >>>>>>>>>>>>>> 1.5.0 but I'm very happy to yield if someone else would be
> > >>>>>>>>>>>>>> interested in
> > >>>>>>>>>>>>>> doing that.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> What do you think?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Aljoscha
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>> Christophe
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
>