(DEPRECATED) Apache Flink Mailing List archive.

[DISCUSS] Drop Bucketing Sink

Classic

List

Threaded

16 messages Options

Robert Metzger

[DISCUSS] Drop Bucketing Sink

Hi all,

I'm currently investigating a failing end to end test for the bucketing
sink [1].
The bucketing sink has been deprecated in the 1.9 release [2], because we
have the new StreamingFileSink [3] for quite a while.
Before putting any effort into fixing the end to end test for the sink, I
wanted to propose dropping the bucketing sink from master for the upcoming
1.11 release.

What do you think?

[1] https://issues.apache.org/jira/browse/FLINK-16227
[2] https://issues.apache.org/jira/browse/FLINK-13396
[3] https://issues.apache.org/jira/browse/FLINK-9749

Kostas Kloudas-4

Re: [DISCUSS] Drop Bucketing Sink

Hi Robert,

+1 for dropping the BucketingSink.
In any case, it has not been maintained for quite some time.

Cheers,
Kostas

On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]> wrote:

>
> Hi all,
>
> I'm currently investigating a failing end to end test for the bucketing
> sink [1].
> The bucketing sink has been deprecated in the 1.9 release [2], because we
> have the new StreamingFileSink [3] for quite a while.
> Before putting any effort into fixing the end to end test for the sink, I
> wanted to propose dropping the bucketing sink from master for the upcoming
> 1.11 release.
>
> What do you think?
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-16227
> [2] https://issues.apache.org/jira/browse/FLINK-13396
> [3] https://issues.apache.org/jira/browse/FLINK-9749

Jeff Zhang

Re: [DISCUSS] Drop Bucketing Sink

+1， dropping deprecated api is always necessary for a sustainable project.

Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：

> Hi Robert,
>
> +1 for dropping the BucketingSink.
> In any case, it has not been maintained for quite some time.
>
> Cheers,
> Kostas
>
> On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]>
> wrote:
> >
> > Hi all,
> >
> > I'm currently investigating a failing end to end test for the bucketing
> > sink [1].
> > The bucketing sink has been deprecated in the 1.9 release [2], because we
> > have the new StreamingFileSink [3] for quite a while.
> > Before putting any effort into fixing the end to end test for the sink, I
> > wanted to propose dropping the bucketing sink from master for the
> upcoming
> > 1.11 release.
> >
> > What do you think?
> >
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > [3] https://issues.apache.org/jira/browse/FLINK-9749
>

--
Best Regards

Jeff Zhang

Aljoscha Krettek-2

Re: [DISCUSS] Drop Bucketing Sink

+1

Aljoscha

Hequn Cheng

Re: [DISCUSS] Drop Bucketing Sink

In reply to this post by Robert Metzger

Good idea! +1 for dropping the BucketingSink.

Best,
Hequn

> On Mar 12, 2020, at 10:40 PM, Robert Metzger <[hidden email]> wrote:
>
> Hi all,
>
> I'm currently investigating a failing end to end test for the bucketing
> sink [1].
> The bucketing sink has been deprecated in the 1.9 release [2], because we
> have the new StreamingFileSink [3] for quite a while.
> Before putting any effort into fixing the end to end test for the sink, I
> wanted to propose dropping the bucketing sink from master for the upcoming
> 1.11 release.
>
> What do you think?
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-16227
> [2] https://issues.apache.org/jira/browse/FLINK-13396
> [3] https://issues.apache.org/jira/browse/FLINK-9749

Zhijiang(wangzhijiang999)

Re: [DISCUSS] Drop Bucketing Sink

In reply to this post by Jeff Zhang

Thanks for driving this discussion, Robert!

This e2e test really fails frequently. +1 to drop bucketing sink, it is not worth paying more efforts since deprecated.

Best,
Zhijiang

------------------------------------------------------------------
From:Jeff Zhang <[hidden email]>
Send Time:2020 Mar. 12 (Thu.) 23:17
To:dev <[hidden email]>
Subject:Re: [DISCUSS] Drop Bucketing Sink

+1， dropping deprecated api is always necessary for a sustainable project.

Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：

--
Best Regards

Jeff Zhang

David Anderson-2

Re: [DISCUSS] Drop Bucketing Sink

The BucketingSink is still somewhat widely used, I think in part because of
shortcomings in the StreamingFileSink.

I would hope that in tandem with removing the bucketing sink we could also
address some of these issues. I'm thinking in particular of issues that are
waiting on FLIP-46 [1].

Removing the bucketing sink will go down better, in my opinion, if it's
coupled with progress on some of the open StreamingFileSink tickets.

Best,
David

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs

On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <[hidden email]>
wrote:

> Thanks for driving this discussion, Robert!
>
> This e2e test really fails frequently. +1 to drop bucketing sink, it is
> not worth paying more efforts since deprecated.
>
> Best,
> Zhijiang
>
>
> ------------------------------------------------------------------
> From:Jeff Zhang <[hidden email]>
> Send Time:2020 Mar. 12 (Thu.) 23:17
> To:dev <[hidden email]>
> Subject:Re: [DISCUSS] Drop Bucketing Sink
>
> +1， dropping deprecated api is always necessary for a sustainable project.
>
> Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
>
> > Hi Robert,
> >
> > +1 for dropping the BucketingSink.
> > In any case, it has not been maintained for quite some time.
> >
> > Cheers,
> > Kostas
> >
> > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]>
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'm currently investigating a failing end to end test for the bucketing
> > > sink [1].
> > > The bucketing sink has been deprecated in the 1.9 release [2], because
> we
> > > have the new StreamingFileSink [3] for quite a while.
> > > Before putting any effort into fixing the end to end test for the
> sink, I
> > > wanted to propose dropping the bucketing sink from master for the
> > upcoming
> > > 1.11 release.
> > >
> > > What do you think?
> > >
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> >
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>

Seth Wiesman-4

Re: [DISCUSS] Drop Bucketing Sink

I agree with David, I think FLIP-49 needs to be prioritized for 1.11 if we
want to drop the bucketing sink.

Seth

On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]> wrote:

> The BucketingSink is still somewhat widely used, I think in part because of
> shortcomings in the StreamingFileSink.
>
> I would hope that in tandem with removing the bucketing sink we could also
> address some of these issues. I'm thinking in particular of issues that are
> waiting on FLIP-46 [1].
>
> Removing the bucketing sink will go down better, in my opinion, if it's
> coupled with progress on some of the open StreamingFileSink tickets.
>
> Best,
> David
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
>
>
> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <[hidden email]
> .invalid>
> wrote:
>
> > Thanks for driving this discussion, Robert!
> >
> > This e2e test really fails frequently. +1 to drop bucketing sink, it is
> > not worth paying more efforts since deprecated.
> >
> > Best,
> > Zhijiang
> >
> >
> > ------------------------------------------------------------------
> > From:Jeff Zhang <[hidden email]>
> > Send Time:2020 Mar. 12 (Thu.) 23:17
> > To:dev <[hidden email]>
> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> >
> > +1， dropping deprecated api is always necessary for a sustainable
> project.
> >
> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> >
> > > Hi Robert,
> > >
> > > +1 for dropping the BucketingSink.
> > > In any case, it has not been maintained for quite some time.
> > >
> > > Cheers,
> > > Kostas
> > >
> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]>
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'm currently investigating a failing end to end test for the
> bucketing
> > > > sink [1].
> > > > The bucketing sink has been deprecated in the 1.9 release [2],
> because
> > we
> > > > have the new StreamingFileSink [3] for quite a while.
> > > > Before putting any effort into fixing the end to end test for the
> > sink, I
> > > > wanted to propose dropping the bucketing sink from master for the
> > > upcoming
> > > > 1.11 release.
> > > >
> > > > What do you think?
> > > >
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> > >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
> >
>

Seth Wiesman-4

Re: [DISCUSS] Drop Bucketing Sink

Sorry, I meant FLIP-46.

Seth

On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]> wrote:

> I agree with David, I think FLIP-49 needs to be prioritized for 1.11 if we
> want to drop the bucketing sink.
>
> Seth
>
> On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]>
> wrote:
>
>> The BucketingSink is still somewhat widely used, I think in part because
>> of
>> shortcomings in the StreamingFileSink.
>>
>> I would hope that in tandem with removing the bucketing sink we could also
>> address some of these issues. I'm thinking in particular of issues that
>> are
>> waiting on FLIP-46 [1].
>>
>> Removing the bucketing sink will go down better, in my opinion, if it's
>> coupled with progress on some of the open StreamingFileSink tickets.
>>
>> Best,
>> David
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
>>
>>
>> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <[hidden email]
>> .invalid>
>> wrote:
>>
>> > Thanks for driving this discussion, Robert!
>> >
>> > This e2e test really fails frequently. +1 to drop bucketing sink, it is
>> > not worth paying more efforts since deprecated.
>> >
>> > Best,
>> > Zhijiang
>> >
>> >
>> > ------------------------------------------------------------------
>> > From:Jeff Zhang <[hidden email]>
>> > Send Time:2020 Mar. 12 (Thu.) 23:17
>> > To:dev <[hidden email]>
>> > Subject:Re: [DISCUSS] Drop Bucketing Sink
>> >
>> > +1， dropping deprecated api is always necessary for a sustainable
>> project.
>> >
>> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
>> >
>> > > Hi Robert,
>> > >
>> > > +1 for dropping the BucketingSink.
>> > > In any case, it has not been maintained for quite some time.
>> > >
>> > > Cheers,
>> > > Kostas
>> > >
>> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]>
>> > > wrote:
>> > > >
>> > > > Hi all,
>> > > >
>> > > > I'm currently investigating a failing end to end test for the
>> bucketing
>> > > > sink [1].
>> > > > The bucketing sink has been deprecated in the 1.9 release [2],
>> because
>> > we
>> > > > have the new StreamingFileSink [3] for quite a while.
>> > > > Before putting any effort into fixing the end to end test for the
>> > sink, I
>> > > > wanted to propose dropping the bucketing sink from master for the
>> > > upcoming
>> > > > 1.11 release.
>> > > >
>> > > > What do you think?
>> > > >
>> > > >
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
>> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
>> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
>> > >
>> >
>> >
>> > --
>> > Best Regards
>> >
>> > Jeff Zhang
>> >
>> >
>>
>

Jingsong Li

Re: [DISCUSS] Drop Bucketing Sink

Hi Robert,

+1 to drop it but maybe not 1.11.

ORC has not been supported on StreamingFileSink. I have seen lots of users
run ORC in the bucketing sink.

Best,
Jingsong Lee

On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[hidden email]> wrote:

> Sorry, I meant FLIP-46.
>
> Seth
>
> On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]> wrote:
>
> > I agree with David, I think FLIP-49 needs to be prioritized for 1.11 if
> we
> > want to drop the bucketing sink.
> >
> > Seth
> >
> > On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]>
> > wrote:
> >
> >> The BucketingSink is still somewhat widely used, I think in part because
> >> of
> >> shortcomings in the StreamingFileSink.
> >>
> >> I would hope that in tandem with removing the bucketing sink we could
> also
> >> address some of these issues. I'm thinking in particular of issues that
> >> are
> >> waiting on FLIP-46 [1].
> >>
> >> Removing the bucketing sink will go down better, in my opinion, if it's
> >> coupled with progress on some of the open StreamingFileSink tickets.
> >>
> >> Best,
> >> David
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> >>
> >>
> >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <[hidden email]
> >> .invalid>
> >> wrote:
> >>
> >> > Thanks for driving this discussion, Robert!
> >> >
> >> > This e2e test really fails frequently. +1 to drop bucketing sink, it
> is
> >> > not worth paying more efforts since deprecated.
> >> >
> >> > Best,
> >> > Zhijiang
> >> >
> >> >
> >> > ------------------------------------------------------------------
> >> > From:Jeff Zhang <[hidden email]>
> >> > Send Time:2020 Mar. 12 (Thu.) 23:17
> >> > To:dev <[hidden email]>
> >> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> >> >
> >> > +1， dropping deprecated api is always necessary for a sustainable
> >> project.
> >> >
> >> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> >> >
> >> > > Hi Robert,
> >> > >
> >> > > +1 for dropping the BucketingSink.
> >> > > In any case, it has not been maintained for quite some time.
> >> > >
> >> > > Cheers,
> >> > > Kostas
> >> > >
> >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]
> >
> >> > > wrote:
> >> > > >
> >> > > > Hi all,
> >> > > >
> >> > > > I'm currently investigating a failing end to end test for the
> >> bucketing
> >> > > > sink [1].
> >> > > > The bucketing sink has been deprecated in the 1.9 release [2],
> >> because
> >> > we
> >> > > > have the new StreamingFileSink [3] for quite a while.
> >> > > > Before putting any effort into fixing the end to end test for the
> >> > sink, I
> >> > > > wanted to propose dropping the bucketing sink from master for the
> >> > > upcoming
> >> > > > 1.11 release.
> >> > > >
> >> > > > What do you think?
> >> > > >
> >> > > >
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> >> > >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> >
> >> > Jeff Zhang
> >> >
> >> >
> >>
> >
>

--
Best, Jingsong Lee

jinhai wang

Re: [DISCUSS] Drop Bucketing Sink

Hi

+1 to drop bucketing sink
FLIP-115 also needs to be prioritized for 1.11

在 2020/3/13 上午10:07，“Jingsong Li”<[hidden email]> 写入:

Hi Robert,

+1 to drop it but maybe not 1.11.

ORC has not been supported on StreamingFileSink. I have seen lots of users
run ORC in the bucketing sink.

Best,
Jingsong Lee

On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[hidden email]> wrote:

> Sorry, I meant FLIP-46.
>
> Seth
>
> On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]> wrote:
>
> > I agree with David, I think FLIP-49 needs to be prioritized for 1.11 if
> we
> > want to drop the bucketing sink.
> >
> > Seth
> >
> > On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]>
> > wrote:
> >
> >> The BucketingSink is still somewhat widely used, I think in part because
> >> of
> >> shortcomings in the StreamingFileSink.
> >>
> >> I would hope that in tandem with removing the bucketing sink we could
> also
> >> address some of these issues. I'm thinking in particular of issues that
> >> are
> >> waiting on FLIP-46 [1].
> >>
> >> Removing the bucketing sink will go down better, in my opinion, if it's
> >> coupled with progress on some of the open StreamingFileSink tickets.
> >>
> >> Best,
> >> David
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> >>
> >>
> >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <[hidden email]
> >> .invalid>
> >> wrote:
> >>
> >> > Thanks for driving this discussion, Robert!
> >> >
> >> > This e2e test really fails frequently. +1 to drop bucketing sink, it
> is
> >> > not worth paying more efforts since deprecated.
> >> >
> >> > Best,
> >> > Zhijiang
> >> >
> >> >
> >> > ------------------------------------------------------------------
> >> > From:Jeff Zhang <[hidden email]>
> >> > Send Time:2020 Mar. 12 (Thu.) 23:17
> >> > To:dev <[hidden email]>
> >> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> >> >
> >> > +1， dropping deprecated api is always necessary for a sustainable
> >> project.
> >> >
> >> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> >> >
> >> > > Hi Robert,
> >> > >
> >> > > +1 for dropping the BucketingSink.
> >> > > In any case, it has not been maintained for quite some time.
> >> > >
> >> > > Cheers,
> >> > > Kostas
> >> > >
> >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <[hidden email]
> >
> >> > > wrote:
> >> > > >
> >> > > > Hi all,
> >> > > >
> >> > > > I'm currently investigating a failing end to end test for the
> >> bucketing
> >> > > > sink [1].
> >> > > > The bucketing sink has been deprecated in the 1.9 release [2],
> >> because
> >> > we
> >> > > > have the new StreamingFileSink [3] for quite a while.
> >> > > > Before putting any effort into fixing the end to end test for the
> >> > sink, I
> >> > > > wanted to propose dropping the bucketing sink from master for the
> >> > > upcoming
> >> > > > 1.11 release.
> >> > > >
> >> > > > What do you think?
> >> > > >
> >> > > >
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> >> > >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> >
> >> > Jeff Zhang
> >> >
> >> >
> >>
> >
>

--
Best, Jingsong Lee

Yu Li

Re: [DISCUSS] Drop Bucketing Sink

In reply to this post by Jingsong Li

Thanks for bringing up this discussion Robert!

According to the inputs, I suggest we create an umbrella JIRA issue to
track all critical improvements StreamingFileSink should have before we
could completely discard bucketing sink, so we could have a clear view of
the progress and how soon we could achieve the goal. What do you think?

btw, we also need to create sub-tasks to track the known issues, for
example it seems I cannot find any JIRA tracking the ORC support yet.

Best Regards,
Yu

On Fri, 13 Mar 2020 at 10:07, Jingsong Li <[hidden email]> wrote:

> Hi Robert,
>
> +1 to drop it but maybe not 1.11.
>
> ORC has not been supported on StreamingFileSink. I have seen lots of users
> run ORC in the bucketing sink.
>
> Best,
> Jingsong Lee
>
> On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[hidden email]> wrote:
>
> > Sorry, I meant FLIP-46.
> >
> > Seth
> >
> > On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]>
> wrote:
> >
> > > I agree with David, I think FLIP-49 needs to be prioritized for 1.11 if
> > we
> > > want to drop the bucketing sink.
> > >
> > > Seth
> > >
> > > On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]>
> > > wrote:
> > >
> > >> The BucketingSink is still somewhat widely used, I think in part
> because
> > >> of
> > >> shortcomings in the StreamingFileSink.
> > >>
> > >> I would hope that in tandem with removing the bucketing sink we could
> > also
> > >> address some of these issues. I'm thinking in particular of issues
> that
> > >> are
> > >> waiting on FLIP-46 [1].
> > >>
> > >> Removing the bucketing sink will go down better, in my opinion, if
> it's
> > >> coupled with progress on some of the open StreamingFileSink tickets.
> > >>
> > >> Best,
> > >> David
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> > >>
> > >>
> > >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <[hidden email]
> > >> .invalid>
> > >> wrote:
> > >>
> > >> > Thanks for driving this discussion, Robert!
> > >> >
> > >> > This e2e test really fails frequently. +1 to drop bucketing sink,
> it
> > is
> > >> > not worth paying more efforts since deprecated.
> > >> >
> > >> > Best,
> > >> > Zhijiang
> > >> >
> > >> >
> > >> > ------------------------------------------------------------------
> > >> > From:Jeff Zhang <[hidden email]>
> > >> > Send Time:2020 Mar. 12 (Thu.) 23:17
> > >> > To:dev <[hidden email]>
> > >> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> > >> >
> > >> > +1， dropping deprecated api is always necessary for a sustainable
> > >> project.
> > >> >
> > >> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> > >> >
> > >> > > Hi Robert,
> > >> > >
> > >> > > +1 for dropping the BucketingSink.
> > >> > > In any case, it has not been maintained for quite some time.
> > >> > >
> > >> > > Cheers,
> > >> > > Kostas
> > >> > >
> > >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <
> [hidden email]
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > Hi all,
> > >> > > >
> > >> > > > I'm currently investigating a failing end to end test for the
> > >> bucketing
> > >> > > > sink [1].
> > >> > > > The bucketing sink has been deprecated in the 1.9 release [2],
> > >> because
> > >> > we
> > >> > > > have the new StreamingFileSink [3] for quite a while.
> > >> > > > Before putting any effort into fixing the end to end test for
> the
> > >> > sink, I
> > >> > > > wanted to propose dropping the bucketing sink from master for
> the
> > >> > > upcoming
> > >> > > > 1.11 release.
> > >> > > >
> > >> > > > What do you think?
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Best Regards
> > >> >
> > >> > Jeff Zhang
> > >> >
> > >> >
> > >>
> > >
> >
>
>
> --
> Best, Jingsong Lee
>

Guowei Ma

Re: [DISCUSS] Drop Bucketing Sink

In reply to this post by Jingsong Li

+1 to drop it.

To Jingsong :
we are planning to implement the orc StreamingFileSink in 1.11.
I think users also could reference the old BucktSink from the old version.

Best,
Guowei

Jingsong Li <[hidden email]> 于2020年3月13日周五上午10:07写道：

Robert Metzger

Re: [DISCUSS] Drop Bucketing Sink

Thank you all for your feedback.

I will try to fix the test then (or disable it).

Here's a ticket for dropping the BucketingSink:
https://issues.apache.org/jira/browse/FLINK-16616 Please mark whatever we
consider necessary as a "depends on" ticket.
@David / @Seth: Where are the tickets depending on FLIP-46 listed? Can one
of you add them to FLINK-16616

On Fri, Mar 13, 2020 at 11:32 AM Guowei Ma <[hidden email]> wrote:

> +1 to drop it.
>
> To Jingsong :
> we are planning to implement the orc StreamingFileSink in 1.11.
> I think users also could reference the old BucktSink from the old version.
>
> Best,
> Guowei
>
>
> Jingsong Li <[hidden email]> 于2020年3月13日周五上午10:07写道：
>
> > Hi Robert,
> >
> > +1 to drop it but maybe not 1.11.
> >
> > ORC has not been supported on StreamingFileSink. I have seen lots of
> users
> > run ORC in the bucketing sink.
> >
> > Best,
> > Jingsong Lee
> >
> > On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[hidden email]>
> wrote:
> >
> > > Sorry, I meant FLIP-46.
> > >
> > > Seth
> > >
> > > On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]>
> > wrote:
> > >
> > > > I agree with David, I think FLIP-49 needs to be prioritized for 1.11
> if
> > > we
> > > > want to drop the bucketing sink.
> > > >
> > > > Seth
> > > >
> > > > On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]
> >
> > > > wrote:
> > > >
> > > >> The BucketingSink is still somewhat widely used, I think in part
> > because
> > > >> of
> > > >> shortcomings in the StreamingFileSink.
> > > >>
> > > >> I would hope that in tandem with removing the bucketing sink we
> could
> > > also
> > > >> address some of these issues. I'm thinking in particular of issues
> > that
> > > >> are
> > > >> waiting on FLIP-46 [1].
> > > >>
> > > >> Removing the bucketing sink will go down better, in my opinion, if
> > it's
> > > >> coupled with progress on some of the open StreamingFileSink tickets.
> > > >>
> > > >> Best,
> > > >> David
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> > > >>
> > > >>
> > > >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <
> [hidden email]
> > > >> .invalid>
> > > >> wrote:
> > > >>
> > > >> > Thanks for driving this discussion, Robert!
> > > >> >
> > > >> > This e2e test really fails frequently. +1 to drop bucketing sink,
> > it
> > > is
> > > >> > not worth paying more efforts since deprecated.
> > > >> >
> > > >> > Best,
> > > >> > Zhijiang
> > > >> >
> > > >> >
> > > >> > ------------------------------------------------------------------
> > > >> > From:Jeff Zhang <[hidden email]>
> > > >> > Send Time:2020 Mar. 12 (Thu.) 23:17
> > > >> > To:dev <[hidden email]>
> > > >> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> > > >> >
> > > >> > +1， dropping deprecated api is always necessary for a sustainable
> > > >> project.
> > > >> >
> > > >> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> > > >> >
> > > >> > > Hi Robert,
> > > >> > >
> > > >> > > +1 for dropping the BucketingSink.
> > > >> > > In any case, it has not been maintained for quite some time.
> > > >> > >
> > > >> > > Cheers,
> > > >> > > Kostas
> > > >> > >
> > > >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <
> > [hidden email]
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > Hi all,
> > > >> > > >
> > > >> > > > I'm currently investigating a failing end to end test for the
> > > >> bucketing
> > > >> > > > sink [1].
> > > >> > > > The bucketing sink has been deprecated in the 1.9 release [2],
> > > >> because
> > > >> > we
> > > >> > > > have the new StreamingFileSink [3] for quite a while.
> > > >> > > > Before putting any effort into fixing the end to end test for
> > the
> > > >> > sink, I
> > > >> > > > wanted to propose dropping the bucketing sink from master for
> > the
> > > >> > > upcoming
> > > >> > > > 1.11 release.
> > > >> > > >
> > > >> > > > What do you think?
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > > >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > > >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Best Regards
> > > >> >
> > > >> > Jeff Zhang
> > > >> >
> > > >> >
> > > >>
> > > >
> > >
> >
> >
> > --
> > Best, Jingsong Lee
> >
>

Kostas Kloudas-4

Re: [DISCUSS] Drop Bucketing Sink

Thanks Robert for all this,

I think that we should also post a thread in the user ML so that users
can also comment on the topic.

What do you think?

Kostas

On Mon, Mar 16, 2020 at 12:27 PM Robert Metzger <[hidden email]> wrote:

>
> Thank you all for your feedback.
>
> I will try to fix the test then (or disable it).
>
> Here's a ticket for dropping the BucketingSink:
> https://issues.apache.org/jira/browse/FLINK-16616 Please mark whatever we
> consider necessary as a "depends on" ticket.
> @David / @Seth: Where are the tickets depending on FLIP-46 listed? Can one
> of you add them to FLINK-16616
>
>
> On Fri, Mar 13, 2020 at 11:32 AM Guowei Ma <[hidden email]> wrote:
>
> > +1 to drop it.
> >
> > To Jingsong :
> > we are planning to implement the orc StreamingFileSink in 1.11.
> > I think users also could reference the old BucktSink from the old version.
> >
> > Best,
> > Guowei
> >
> >
> > Jingsong Li <[hidden email]> 于2020年3月13日周五上午10:07写道：
> >
> > > Hi Robert,
> > >
> > > +1 to drop it but maybe not 1.11.
> > >
> > > ORC has not been supported on StreamingFileSink. I have seen lots of
> > users
> > > run ORC in the bucketing sink.
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > > On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[hidden email]>
> > wrote:
> > >
> > > > Sorry, I meant FLIP-46.
> > > >
> > > > Seth
> > > >
> > > > On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]>
> > > wrote:
> > > >
> > > > > I agree with David, I think FLIP-49 needs to be prioritized for 1.11
> > if
> > > > we
> > > > > want to drop the bucketing sink.
> > > > >
> > > > > Seth
> > > > >
> > > > > On Thu, Mar 12, 2020 at 10:53 AM David Anderson <[hidden email]
> > >
> > > > > wrote:
> > > > >
> > > > >> The BucketingSink is still somewhat widely used, I think in part
> > > because
> > > > >> of
> > > > >> shortcomings in the StreamingFileSink.
> > > > >>
> > > > >> I would hope that in tandem with removing the bucketing sink we
> > could
> > > > also
> > > > >> address some of these issues. I'm thinking in particular of issues
> > > that
> > > > >> are
> > > > >> waiting on FLIP-46 [1].
> > > > >>
> > > > >> Removing the bucketing sink will go down better, in my opinion, if
> > > it's
> > > > >> coupled with progress on some of the open StreamingFileSink tickets.
> > > > >>
> > > > >> Best,
> > > > >> David
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> > > > >>
> > > > >>
> > > > >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <
> > [hidden email]
> > > > >> .invalid>
> > > > >> wrote:
> > > > >>
> > > > >> > Thanks for driving this discussion, Robert!
> > > > >> >
> > > > >> > This e2e test really fails frequently. +1 to drop bucketing sink,
> > > it
> > > > is
> > > > >> > not worth paying more efforts since deprecated.
> > > > >> >
> > > > >> > Best,
> > > > >> > Zhijiang
> > > > >> >
> > > > >> >
> > > > >> > ------------------------------------------------------------------
> > > > >> > From:Jeff Zhang <[hidden email]>
> > > > >> > Send Time:2020 Mar. 12 (Thu.) 23:17
> > > > >> > To:dev <[hidden email]>
> > > > >> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> > > > >> >
> > > > >> > +1， dropping deprecated api is always necessary for a sustainable
> > > > >> project.
> > > > >> >
> > > > >> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> > > > >> >
> > > > >> > > Hi Robert,
> > > > >> > >
> > > > >> > > +1 for dropping the BucketingSink.
> > > > >> > > In any case, it has not been maintained for quite some time.
> > > > >> > >
> > > > >> > > Cheers,
> > > > >> > > Kostas
> > > > >> > >
> > > > >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <
> > > [hidden email]
> > > > >
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > Hi all,
> > > > >> > > >
> > > > >> > > > I'm currently investigating a failing end to end test for the
> > > > >> bucketing
> > > > >> > > > sink [1].
> > > > >> > > > The bucketing sink has been deprecated in the 1.9 release [2],
> > > > >> because
> > > > >> > we
> > > > >> > > > have the new StreamingFileSink [3] for quite a while.
> > > > >> > > > Before putting any effort into fixing the end to end test for
> > > the
> > > > >> > sink, I
> > > > >> > > > wanted to propose dropping the bucketing sink from master for
> > > the
> > > > >> > > upcoming
> > > > >> > > > 1.11 release.
> > > > >> > > >
> > > > >> > > > What do you think?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > > > >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > > > >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Best Regards
> > > > >> >
> > > > >> > Jeff Zhang
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best, Jingsong Lee
> > >
> >

Robert Metzger

Re: [DISCUSS] Drop Bucketing Sink

By the way: I have disabled the test in the Hadoop 2.4.1 build, so my
original problem is resolved.

I'm not convinced that posting on the user@ ml is the right approach.
The last few questions to the user@ list asking for feedback have not
really been answered. I believe that we have deprecated the sink for long
enough, also we seem to have a common understanding of missing features we
need to resolve first.
Last but not least, we can always port the bucketing sink to another
repository (apache bahir) and keep compatibility with later Flink versions
there if a lot of users complain once it has been removed.

On Tue, Mar 17, 2020 at 10:35 AM Kostas Kloudas <[hidden email]> wrote:

> Thanks Robert for all this,
>
> I think that we should also post a thread in the user ML so that users
> can also comment on the topic.
>
> What do you think?
>
> Kostas
>
> On Mon, Mar 16, 2020 at 12:27 PM Robert Metzger <[hidden email]>
> wrote:
> >
> > Thank you all for your feedback.
> >
> > I will try to fix the test then (or disable it).
> >
> > Here's a ticket for dropping the BucketingSink:
> > https://issues.apache.org/jira/browse/FLINK-16616 Please mark whatever
> we
> > consider necessary as a "depends on" ticket.
> > @David / @Seth: Where are the tickets depending on FLIP-46 listed? Can
> one
> > of you add them to FLINK-16616
> >
> >
> > On Fri, Mar 13, 2020 at 11:32 AM Guowei Ma <[hidden email]> wrote:
> >
> > > +1 to drop it.
> > >
> > > To Jingsong :
> > > we are planning to implement the orc StreamingFileSink in 1.11.
> > > I think users also could reference the old BucktSink from the old
> version.
> > >
> > > Best,
> > > Guowei
> > >
> > >
> > > Jingsong Li <[hidden email]> 于2020年3月13日周五上午10:07写道：
> > >
> > > > Hi Robert,
> > > >
> > > > +1 to drop it but maybe not 1.11.
> > > >
> > > > ORC has not been supported on StreamingFileSink. I have seen lots of
> > > users
> > > > run ORC in the bucketing sink.
> > > >
> > > > Best,
> > > > Jingsong Lee
> > > >
> > > > On Fri, Mar 13, 2020 at 1:11 AM Seth Wiesman <[hidden email]>
> > > wrote:
> > > >
> > > > > Sorry, I meant FLIP-46.
> > > > >
> > > > > Seth
> > > > >
> > > > > On Thu, Mar 12, 2020 at 11:52 AM Seth Wiesman <[hidden email]
> >
> > > > wrote:
> > > > >
> > > > > > I agree with David, I think FLIP-49 needs to be prioritized for
> 1.11
> > > if
> > > > > we
> > > > > > want to drop the bucketing sink.
> > > > > >
> > > > > > Seth
> > > > > >
> > > > > > On Thu, Mar 12, 2020 at 10:53 AM David Anderson <
> [hidden email]
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> The BucketingSink is still somewhat widely used, I think in part
> > > > because
> > > > > >> of
> > > > > >> shortcomings in the StreamingFileSink.
> > > > > >>
> > > > > >> I would hope that in tandem with removing the bucketing sink we
> > > could
> > > > > also
> > > > > >> address some of these issues. I'm thinking in particular of
> issues
> > > > that
> > > > > >> are
> > > > > >> waiting on FLIP-46 [1].
> > > > > >>
> > > > > >> Removing the bucketing sink will go down better, in my opinion,
> if
> > > > it's
> > > > > >> coupled with progress on some of the open StreamingFileSink
> tickets.
> > > > > >>
> > > > > >> Best,
> > > > > >> David
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-46%3A+Graceful+Shutdown+Handling+by+UDFs
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Mar 12, 2020 at 4:27 PM Zhijiang <
> > > [hidden email]
> > > > > >> .invalid>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Thanks for driving this discussion, Robert!
> > > > > >> >
> > > > > >> > This e2e test really fails frequently. +1 to drop bucketing
> sink,
> > > > it
> > > > > is
> > > > > >> > not worth paying more efforts since deprecated.
> > > > > >> >
> > > > > >> > Best,
> > > > > >> > Zhijiang
> > > > > >> >
> > > > > >> >
> > > > > >> >
> ------------------------------------------------------------------
> > > > > >> > From:Jeff Zhang <[hidden email]>
> > > > > >> > Send Time:2020 Mar. 12 (Thu.) 23:17
> > > > > >> > To:dev <[hidden email]>
> > > > > >> > Subject:Re: [DISCUSS] Drop Bucketing Sink
> > > > > >> >
> > > > > >> > +1， dropping deprecated api is always necessary for a
> sustainable
> > > > > >> project.
> > > > > >> >
> > > > > >> > Kostas Kloudas <[hidden email]> 于2020年3月12日周四下午11:06写道：
> > > > > >> >
> > > > > >> > > Hi Robert,
> > > > > >> > >
> > > > > >> > > +1 for dropping the BucketingSink.
> > > > > >> > > In any case, it has not been maintained for quite some time.
> > > > > >> > >
> > > > > >> > > Cheers,
> > > > > >> > > Kostas
> > > > > >> > >
> > > > > >> > > On Thu, Mar 12, 2020 at 3:41 PM Robert Metzger <
> > > > [hidden email]
> > > > > >
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > Hi all,
> > > > > >> > > >
> > > > > >> > > > I'm currently investigating a failing end to end test for
> the
> > > > > >> bucketing
> > > > > >> > > > sink [1].
> > > > > >> > > > The bucketing sink has been deprecated in the 1.9 release
> [2],
> > > > > >> because
> > > > > >> > we
> > > > > >> > > > have the new StreamingFileSink [3] for quite a while.
> > > > > >> > > > Before putting any effort into fixing the end to end test
> for
> > > > the
> > > > > >> > sink, I
> > > > > >> > > > wanted to propose dropping the bucketing sink from master
> for
> > > > the
> > > > > >> > > upcoming
> > > > > >> > > > 1.11 release.
> > > > > >> > > >
> > > > > >> > > > What do you think?
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > [1] https://issues.apache.org/jira/browse/FLINK-16227
> > > > > >> > > > [2] https://issues.apache.org/jira/browse/FLINK-13396
> > > > > >> > > > [3] https://issues.apache.org/jira/browse/FLINK-9749
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Best Regards
> > > > > >> >
> > > > > >> > Jeff Zhang
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best, Jingsong Lee
> > > >
> > >
>