Cherry picking FLINK-9753 and FLINK-9753 on release-1.6

[Kostas Kloudas]

Hi all,

We just merged on the master [FLINK-9753] (commit 66b1f854a0250bdd048808d40f93aa2990476841)
and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).

These two commits introduce:
1) a big refactoring to the new StreamingFileSink which allows it to support bulk formats in general (FLINK-9753) and more expressive rolling and bucketing strategies, and
2) support for Parque, based on the previous refactoring.

Both these features were pretty popular in the mailing list.

Given that these 2 features are self contained, as they mainly affect a connector,  we would like to cherry pick these commits to the release-1.6 branch. But for this, we would like to have the OK from the community.

Please let me know what you think,
Kostas

https://issues.apache.org/jira/browse/FLINK-9903 <https://issues.apache.org/jira/browse/FLINK-9903>
https://issues.apache.org/jira/browse/FLINK-9753 <https://issues.apache.org/jira/browse/FLINK-9753>

[魏偉哲]

Hi Kostas,

Nice to see these features merged to the release-1.6 branch. Supporting
Parque format in StreamingFileSink
really makes me eager to apply it to some use cases in our company, but I'm
still wondering if the current
StreamingFileSink works well on S3 or I should wait util [FLINK-9752] done.

Anyway, I would still like to see this happen, because it let me have more
alternative to upload data with parquet
format. Thanks for your great efforts on this.

Best Regards,
Tony Wei

https://issues.apache.org/jira/browse/FLINK-9752

2018-07-21 4:24 GMT+08:00 Kostas Kloudas <[hidden email]>:

> Hi all,
>
> We just merged on the master [FLINK-9753] (commit
> 66b1f854a0250bdd048808d40f93aa2990476841)
> and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
>
> These two commits introduce:
> 1) a big refactoring to the new StreamingFileSink which allows it to
> support bulk formats in general (FLINK-9753) and more expressive rolling
> and bucketing strategies, and
> 2) support for Parque, based on the previous refactoring.
>
> Both these features were pretty popular in the mailing list.
>
> Given that these 2 features are self contained, as they mainly affect a
> connector,  we would like to cherry pick these commits to the release-1.6
> branch. But for this, we would like to have the OK from the community.
>
> Please let me know what you think,
> Kostas
>
> https://issues.apache.org/jira/browse/FLINK-9903 <
> https://issues.apache.org/jira/browse/FLINK-9903>
> https://issues.apache.org/jira/browse/FLINK-9753 <
> https://issues.apache.org/jira/browse/FLINK-9753>

[Till Rohrmann]

I would be in favor of including FLINK-9903 and FLINK-9753 in Flink 1.6
because they add more value for the user at a comparable small risk of
breaking the system since it is a connector. So +1 for cherry picking these
two commits.

Cheers,
Till

On Sat, Jul 21, 2018 at 6:16 PM Tony Wei <[hidden email]> wrote:

> Hi Kostas,
>
> Nice to see these features merged to the release-1.6 branch. Supporting
> Parque format in StreamingFileSink
> really makes me eager to apply it to some use cases in our company, but I'm
> still wondering if the current
> StreamingFileSink works well on S3 or I should wait util [FLINK-9752] done.
>
> Anyway, I would still like to see this happen, because it let me have more
> alternative to upload data with parquet
> format. Thanks for your great efforts on this.
>
> Best Regards,
> Tony Wei
>
> https://issues.apache.org/jira/browse/FLINK-9752
>
> 2018-07-21 4:24 GMT+08:00 Kostas Kloudas <[hidden email]>:
>
> > Hi all,
> >
> > We just merged on the master [FLINK-9753] (commit
> > 66b1f854a0250bdd048808d40f93aa2990476841)
> > and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
> >
> > These two commits introduce:
> > 1) a big refactoring to the new StreamingFileSink which allows it to
> > support bulk formats in general (FLINK-9753) and more expressive rolling
> > and bucketing strategies, and
> > 2) support for Parque, based on the previous refactoring.
> >
> > Both these features were pretty popular in the mailing list.
> >
> > Given that these 2 features are self contained, as they mainly affect a
> > connector,  we would like to cherry pick these commits to the release-1.6
> > branch. But for this, we would like to have the OK from the community.
> >
> > Please let me know what you think,
> > Kostas
> >
> > https://issues.apache.org/jira/browse/FLINK-9903 <
> > https://issues.apache.org/jira/browse/FLINK-9903>
> > https://issues.apache.org/jira/browse/FLINK-9753 <
> > https://issues.apache.org/jira/browse/FLINK-9753>
>

[Kostas Kloudas]

Thanks for the +1’s.

I added the relevant commits to the release-1.6 branch.

Cheers,
Kostas

> On Jul 22, 2018, at 3:05 PM, Till Rohrmann <[hidden email]> wrote:
>
> I would be in favor of including FLINK-9903 and FLINK-9753 in Flink 1.6
> because they add more value for the user at a comparable small risk of
> breaking the system since it is a connector. So +1 for cherry picking these
> two commits.
>
> Cheers,
> Till
>
> On Sat, Jul 21, 2018 at 6:16 PM Tony Wei <[hidden email]> wrote:
>
>> Hi Kostas,
>>
>> Nice to see these features merged to the release-1.6 branch. Supporting
>> Parque format in StreamingFileSink
>> really makes me eager to apply it to some use cases in our company, but I'm
>> still wondering if the current
>> StreamingFileSink works well on S3 or I should wait util [FLINK-9752] done.
>>
>> Anyway, I would still like to see this happen, because it let me have more
>> alternative to upload data with parquet
>> format. Thanks for your great efforts on this.
>>
>> Best Regards,
>> Tony Wei
>>
>> https://issues.apache.org/jira/browse/FLINK-9752
>>
>> 2018-07-21 4:24 GMT+08:00 Kostas Kloudas <[hidden email]>:
>>
>>> Hi all,
>>>
>>> We just merged on the master [FLINK-9753] (commit
>>> 66b1f854a0250bdd048808d40f93aa2990476841)
>>> and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
>>>
>>> These two commits introduce:
>>> 1) a big refactoring to the new StreamingFileSink which allows it to
>>> support bulk formats in general (FLINK-9753) and more expressive rolling
>>> and bucketing strategies, and
>>> 2) support for Parque, based on the previous refactoring.
>>>
>>> Both these features were pretty popular in the mailing list.
>>>
>>> Given that these 2 features are self contained, as they mainly affect a
>>> connector,  we would like to cherry pick these commits to the release-1.6
>>> branch. But for this, we would like to have the OK from the community.
>>>
>>> Please let me know what you think,
>>> Kostas
>>>
>>> https://issues.apache.org/jira/browse/FLINK-9903 <
>>> https://issues.apache.org/jira/browse/FLINK-9903>
>>> https://issues.apache.org/jira/browse/FLINK-9753 <
>>> https://issues.apache.org/jira/browse/FLINK-9753>
>>

[vino yang]

I also see some file sink questions, so +1 for picking the PR about
StreamingFileSink.

Thanks, vino.

2018-07-23 20:53 GMT+08:00 Kostas Kloudas <[hidden email]>:

> Thanks for the +1’s.
>
> I added the relevant commits to the release-1.6 branch.
>
> Cheers,
> Kostas
>
> > On Jul 22, 2018, at 3:05 PM, Till Rohrmann <[hidden email]> wrote:
> >
> > I would be in favor of including FLINK-9903 and FLINK-9753 in Flink 1.6
> > because they add more value for the user at a comparable small risk of
> > breaking the system since it is a connector. So +1 for cherry picking
> these
> > two commits.
> >
> > Cheers,
> > Till
> >
> > On Sat, Jul 21, 2018 at 6:16 PM Tony Wei <[hidden email]> wrote:
> >
> >> Hi Kostas,
> >>
> >> Nice to see these features merged to the release-1.6 branch. Supporting
> >> Parque format in StreamingFileSink
> >> really makes me eager to apply it to some use cases in our company, but
> I'm
> >> still wondering if the current
> >> StreamingFileSink works well on S3 or I should wait util [FLINK-9752]
> done.
> >>
> >> Anyway, I would still like to see this happen, because it let me have
> more
> >> alternative to upload data with parquet
> >> format. Thanks for your great efforts on this.
> >>
> >> Best Regards,
> >> Tony Wei
> >>
> >> https://issues.apache.org/jira/browse/FLINK-9752
> >>
> >> 2018-07-21 4:24 GMT+08:00 Kostas Kloudas <[hidden email]>:
> >>
> >>> Hi all,
> >>>
> >>> We just merged on the master [FLINK-9753] (commit
> >>> 66b1f854a0250bdd048808d40f93aa2990476841)
> >>> and [FLINK-9903] (commit b56c75ca375049b1d2c80d2d0945ae1ae04eb39e).
> >>>
> >>> These two commits introduce:
> >>> 1) a big refactoring to the new StreamingFileSink which allows it to
> >>> support bulk formats in general (FLINK-9753) and more expressive
> rolling
> >>> and bucketing strategies, and
> >>> 2) support for Parque, based on the previous refactoring.
> >>>
> >>> Both these features were pretty popular in the mailing list.
> >>>
> >>> Given that these 2 features are self contained, as they mainly affect a
> >>> connector,  we would like to cherry pick these commits to the
> release-1.6
> >>> branch. But for this, we would like to have the OK from the community.
> >>>
> >>> Please let me know what you think,
> >>> Kostas
> >>>
> >>> https://issues.apache.org/jira/browse/FLINK-9903 <
> >>> https://issues.apache.org/jira/browse/FLINK-9903>
> >>> https://issues.apache.org/jira/browse/FLINK-9753 <
> >>> https://issues.apache.org/jira/browse/FLINK-9753>
> >>
>
>