[DISCUSS] Upgrade kinesis connector to Apache 2.0 License and include it in official release

[bowen.li]

Hi all,

A while back we discussed upgrading flink-connector-kinesis module to
Apache 2.0 license so that its jar can be published as part of official
Flink releases. Given we have a large user base using Flink with
kinesis/dynamodb streams, it'll free users from building and maintaining
the module themselves, and improve user and developer experience. A ticket
was created [1] but has been idle mainly due to new releases of some aws
libs are not available yet then.

As of today I see that all flink-connector-kinesis's aws dependencies have
been updated to Apache 2.0 license and are released. They include:

- aws-java-sdk-kinesis
- aws-java-sdk-sts
- amazon-kinesis-client
- amazon-kinesis-producer (Apache 2.0 from 0.13.1, released 18 days ago) [2]
- dynamodb-streams-kinesis-adapter (Apache 2.0 from 1.5.0, released 7 days
ago) [3]

Therefore, I'd suggest we kick off the initiative and aim for release 1.10
which is roughly 3 months away, leaving us plenty of time to finish.
According to @Dyana 's comment in the ticket [1], seems some large chunks
of changes need to be made into multiple parts than simply upgrading lib
versions, so we can further break the JIRA down into sub-tasks to limit
scope of each change for easier code review.

@Dyana would you still be interested in carrying the responsibility and
forwarding the effort?

Thanks,
Bowen

[1] https://issues.apache.org/jira/browse/FLINK-12847
[2] https://github.com/awslabs/amazon-kinesis-producer/releases
[3] https://github.com/awslabs/dynamodb-streams-kinesis-adapter/releases

[Dyana Rose]

Ahh, brilliant, I had myself on notifications for the streams adapter
releases, but must have missed it. That's great news.

I've got the branch prepped for moving over to Apache 2.0, but staying on
KCL 1.x, which requires the least amount of change.

Considering the large amount of change required to update to KCL/SDK 2.x I
would recommend that be done in a parallel task. Making both connectors
available then for usage, 1.x and 2.x. If that makes sense.

The branch I push will have the English Language documents updated, but not
have the Chinese Language documents updated. Is there a process for this?

Thanks,
Dyana

On Mon, 19 Aug 2019 at 19:08, Bowen Li <[hidden email]> wrote:

> Hi all,
>
> A while back we discussed upgrading flink-connector-kinesis module to
> Apache 2.0 license so that its jar can be published as part of official
> Flink releases. Given we have a large user base using Flink with
> kinesis/dynamodb streams, it'll free users from building and maintaining
> the module themselves, and improve user and developer experience. A ticket
> was created [1] but has been idle mainly due to new releases of some aws
> libs are not available yet then.
>
> As of today I see that all flink-connector-kinesis's aws dependencies have
> been updated to Apache 2.0 license and are released. They include:
>
> - aws-java-sdk-kinesis
> - aws-java-sdk-sts
> - amazon-kinesis-client
> - amazon-kinesis-producer (Apache 2.0 from 0.13.1, released 18 days ago)
> [2]
> - dynamodb-streams-kinesis-adapter (Apache 2.0 from 1.5.0, released 7 days
> ago) [3]
>
> Therefore, I'd suggest we kick off the initiative and aim for release 1.10
> which is roughly 3 months away, leaving us plenty of time to finish.
> According to @Dyana 's comment in the ticket [1], seems some large chunks
> of changes need to be made into multiple parts than simply upgrading lib
> versions, so we can further break the JIRA down into sub-tasks to limit
> scope of each change for easier code review.
>
> @Dyana would you still be interested in carrying the responsibility and
> forwarding the effort?
>
> Thanks,
> Bowen
>
> [1] https://issues.apache.org/jira/browse/FLINK-12847
> [2] https://github.com/awslabs/amazon-kinesis-producer/releases
> [3] https://github.com/awslabs/dynamodb-streams-kinesis-adapter/releases
>
>
>

--

Dyana Rose
Software Engineer


W: www.salecycle.com <http://www.salecycle.com/>
[image: Airline & Travel Booking Trends - Download Report]
<https://t.xink.io/Tracking/Index/WtEBAKNtAAAwphkA0>

[Stephan Ewen]

Just FYI - Becket, Aljoscha, and me are working on fleshing out the
remaining details of FLIP-27 (new source API).
We will share this as soon as we have made some progress on some of the
details.

The Kinesis connector would be one of the first that we would try to also
implement in that new API, as a validation that it is powerful and flexible
enough.

If the upgrade involved major refactoring, would it make sense combine
these efforts?

Best,
Stephan


On Tue, Aug 20, 2019 at 9:12 AM Dyana Rose <[hidden email]> wrote:

> Ahh, brilliant, I had myself on notifications for the streams adapter
> releases, but must have missed it. That's great news.
>
> I've got the branch prepped for moving over to Apache 2.0, but staying on
> KCL 1.x, which requires the least amount of change.
>
> Considering the large amount of change required to update to KCL/SDK 2.x I
> would recommend that be done in a parallel task. Making both connectors
> available then for usage, 1.x and 2.x. If that makes sense.
>
> The branch I push will have the English Language documents updated, but not
> have the Chinese Language documents updated. Is there a process for this?
>
> Thanks,
> Dyana
>
> On Mon, 19 Aug 2019 at 19:08, Bowen Li <[hidden email]> wrote:
>
> > Hi all,
> >
> > A while back we discussed upgrading flink-connector-kinesis module to
> > Apache 2.0 license so that its jar can be published as part of official
> > Flink releases. Given we have a large user base using Flink with
> > kinesis/dynamodb streams, it'll free users from building and maintaining
> > the module themselves, and improve user and developer experience. A
> ticket
> > was created [1] but has been idle mainly due to new releases of some aws
> > libs are not available yet then.
> >
> > As of today I see that all flink-connector-kinesis's aws dependencies
> have
> > been updated to Apache 2.0 license and are released. They include:
> >
> > - aws-java-sdk-kinesis
> > - aws-java-sdk-sts
> > - amazon-kinesis-client
> > - amazon-kinesis-producer (Apache 2.0 from 0.13.1, released 18 days ago)
> > [2]
> > - dynamodb-streams-kinesis-adapter (Apache 2.0 from 1.5.0, released 7
> days
> > ago) [3]
> >
> > Therefore, I'd suggest we kick off the initiative and aim for release
> 1.10
> > which is roughly 3 months away, leaving us plenty of time to finish.
> > According to @Dyana 's comment in the ticket [1], seems some large chunks
> > of changes need to be made into multiple parts than simply upgrading lib
> > versions, so we can further break the JIRA down into sub-tasks to limit
> > scope of each change for easier code review.
> >
> > @Dyana would you still be interested in carrying the responsibility and
> > forwarding the effort?
> >
> > Thanks,
> > Bowen
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-12847
> > [2] https://github.com/awslabs/amazon-kinesis-producer/releases
> > [3] https://github.com/awslabs/dynamodb-streams-kinesis-adapter/releases
> >
> >
> >
>
> --
>
> Dyana Rose
> Software Engineer
>
>
> W: www.salecycle.com <http://www.salecycle.com/>
> [image: Airline & Travel Booking Trends - Download Report]
> <https://t.xink.io/Tracking/Index/WtEBAKNtAAAwphkA0>
>

[Becket Qin]

I agree with Stephan. It will be good to see if we can align those two
efforts so that we don't write code that are soon to be refactored again.

Thanks,

Jiangjie (Becket) Qin

On Tue, Aug 20, 2019 at 10:50 AM Stephan Ewen <[hidden email]> wrote:

> Just FYI - Becket, Aljoscha, and me are working on fleshing out the
> remaining details of FLIP-27 (new source API).
> We will share this as soon as we have made some progress on some of the
> details.
>
> The Kinesis connector would be one of the first that we would try to also
> implement in that new API, as a validation that it is powerful and flexible
> enough.
>
> If the upgrade involved major refactoring, would it make sense combine
> these efforts?
>
> Best,
> Stephan
>
>
> On Tue, Aug 20, 2019 at 9:12 AM Dyana Rose <[hidden email]>
> wrote:
>
> > Ahh, brilliant, I had myself on notifications for the streams adapter
> > releases, but must have missed it. That's great news.
> >
> > I've got the branch prepped for moving over to Apache 2.0, but staying on
> > KCL 1.x, which requires the least amount of change.
> >
> > Considering the large amount of change required to update to KCL/SDK 2.x
> I
> > would recommend that be done in a parallel task. Making both connectors
> > available then for usage, 1.x and 2.x. If that makes sense.
> >
> > The branch I push will have the English Language documents updated, but
> not
> > have the Chinese Language documents updated. Is there a process for this?
> >
> > Thanks,
> > Dyana
> >
> > On Mon, 19 Aug 2019 at 19:08, Bowen Li <[hidden email]> wrote:
> >
> > > Hi all,
> > >
> > > A while back we discussed upgrading flink-connector-kinesis module to
> > > Apache 2.0 license so that its jar can be published as part of official
> > > Flink releases. Given we have a large user base using Flink with
> > > kinesis/dynamodb streams, it'll free users from building and
> maintaining
> > > the module themselves, and improve user and developer experience. A
> > ticket
> > > was created [1] but has been idle mainly due to new releases of some
> aws
> > > libs are not available yet then.
> > >
> > > As of today I see that all flink-connector-kinesis's aws dependencies
> > have
> > > been updated to Apache 2.0 license and are released. They include:
> > >
> > > - aws-java-sdk-kinesis
> > > - aws-java-sdk-sts
> > > - amazon-kinesis-client
> > > - amazon-kinesis-producer (Apache 2.0 from 0.13.1, released 18 days
> ago)
> > > [2]
> > > - dynamodb-streams-kinesis-adapter (Apache 2.0 from 1.5.0, released 7
> > days
> > > ago) [3]
> > >
> > > Therefore, I'd suggest we kick off the initiative and aim for release
> > 1.10
> > > which is roughly 3 months away, leaving us plenty of time to finish.
> > > According to @Dyana 's comment in the ticket [1], seems some large
> chunks
> > > of changes need to be made into multiple parts than simply upgrading
> lib
> > > versions, so we can further break the JIRA down into sub-tasks to limit
> > > scope of each change for easier code review.
> > >
> > > @Dyana would you still be interested in carrying the responsibility and
> > > forwarding the effort?
> > >
> > > Thanks,
> > > Bowen
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-12847
> > > [2] https://github.com/awslabs/amazon-kinesis-producer/releases
> > > [3]
> https://github.com/awslabs/dynamodb-streams-kinesis-adapter/releases
> > >
> > >
> > >
> >
> > --
> >
> > Dyana Rose
> > Software Engineer
> >
> >
> > W: www.salecycle.com <http://www.salecycle.com/>
> > [image: Airline & Travel Booking Trends - Download Report]
> > <https://t.xink.io/Tracking/Index/WtEBAKNtAAAwphkA0>
> >
>

[Tzu-Li (Gordon) Tai]

Hi Dyana,

Regarding your question on the Chinese docs:
Since the Chinese counterparts for the Kinesis connector documentation
isn't translated yet (see docs/dev/connectors/kinesis.zh.md), for now you
can simply just sync whatever changes you made to the English doc to the
Chinese one as well.

Cheers,
Gordon

On Tue, Aug 20, 2019 at 1:22 PM Becket Qin <[hidden email]> wrote:

> I agree with Stephan. It will be good to see if we can align those two
> efforts so that we don't write code that are soon to be refactored again.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Aug 20, 2019 at 10:50 AM Stephan Ewen <[hidden email]> wrote:
>
> > Just FYI - Becket, Aljoscha, and me are working on fleshing out the
> > remaining details of FLIP-27 (new source API).
> > We will share this as soon as we have made some progress on some of the
> > details.
> >
> > The Kinesis connector would be one of the first that we would try to also
> > implement in that new API, as a validation that it is powerful and
> flexible
> > enough.
> >
> > If the upgrade involved major refactoring, would it make sense combine
> > these efforts?
> >
> > Best,
> > Stephan
> >
> >
> > On Tue, Aug 20, 2019 at 9:12 AM Dyana Rose <[hidden email]>
> > wrote:
> >
> > > Ahh, brilliant, I had myself on notifications for the streams adapter
> > > releases, but must have missed it. That's great news.
> > >
> > > I've got the branch prepped for moving over to Apache 2.0, but staying
> on
> > > KCL 1.x, which requires the least amount of change.
> > >
> > > Considering the large amount of change required to update to KCL/SDK
> 2.x
> > I
> > > would recommend that be done in a parallel task. Making both connectors
> > > available then for usage, 1.x and 2.x. If that makes sense.
> > >
> > > The branch I push will have the English Language documents updated, but
> > not
> > > have the Chinese Language documents updated. Is there a process for
> this?
> > >
> > > Thanks,
> > > Dyana
> > >
> > > On Mon, 19 Aug 2019 at 19:08, Bowen Li <[hidden email]> wrote:
> > >
> > > > Hi all,
> > > >
> > > > A while back we discussed upgrading flink-connector-kinesis module to
> > > > Apache 2.0 license so that its jar can be published as part of
> official
> > > > Flink releases. Given we have a large user base using Flink with
> > > > kinesis/dynamodb streams, it'll free users from building and
> > maintaining
> > > > the module themselves, and improve user and developer experience. A
> > > ticket
> > > > was created [1] but has been idle mainly due to new releases of some
> > aws
> > > > libs are not available yet then.
> > > >
> > > > As of today I see that all flink-connector-kinesis's aws dependencies
> > > have
> > > > been updated to Apache 2.0 license and are released. They include:
> > > >
> > > > - aws-java-sdk-kinesis
> > > > - aws-java-sdk-sts
> > > > - amazon-kinesis-client
> > > > - amazon-kinesis-producer (Apache 2.0 from 0.13.1, released 18 days
> > ago)
> > > > [2]
> > > > - dynamodb-streams-kinesis-adapter (Apache 2.0 from 1.5.0, released 7
> > > days
> > > > ago) [3]
> > > >
> > > > Therefore, I'd suggest we kick off the initiative and aim for release
> > > 1.10
> > > > which is roughly 3 months away, leaving us plenty of time to finish.
> > > > According to @Dyana 's comment in the ticket [1], seems some large
> > chunks
> > > > of changes need to be made into multiple parts than simply upgrading
> > lib
> > > > versions, so we can further break the JIRA down into sub-tasks to
> limit
> > > > scope of each change for easier code review.
> > > >
> > > > @Dyana would you still be interested in carrying the responsibility
> and
> > > > forwarding the effort?
> > > >
> > > > Thanks,
> > > > Bowen
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-12847
> > > > [2] https://github.com/awslabs/amazon-kinesis-producer/releases
> > > > [3]
> > https://github.com/awslabs/dynamodb-streams-kinesis-adapter/releases
> > > >
> > > >
> > > >
> > >
> > > --
> > >
> > > Dyana Rose
> > > Software Engineer
> > >
> > >
> > > W: www.salecycle.com <http://www.salecycle.com/>
> > > [image: Airline & Travel Booking Trends - Download Report]
> > > <https://t.xink.io/Tracking/Index/WtEBAKNtAAAwphkA0>
> > >
> >
>

[Dyana Rose]

ok great,

that's done, the PR is rebased and squashed on top of master and is running
through Travis

https://github.com/apache/flink/pull/9494

Dyana

On Tue, 20 Aug 2019 at 15:32, Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> Hi Dyana,
>
> Regarding your question on the Chinese docs:
> Since the Chinese counterparts for the Kinesis connector documentation
> isn't translated yet (see docs/dev/connectors/kinesis.zh.md), for now you
> can simply just sync whatever changes you made to the English doc to the
> Chinese one as well.
>
> Cheers,
> Gordon
>

[bowen.li]

@Stephan @Becket kinesis connector currently is using KCL 1.9. Mass changes
are needed if we switch to KCL 2.x. I agree with Dyana that, since KCL 1.x
is also updated to Apache 2.0, we can just focus on upgrading to a newer
KCL 1.x minor version for now.

On Tue, Aug 20, 2019 at 7:52 AM Dyana Rose <[hidden email]> wrote:

> ok great,
>
> that's done, the PR is rebased and squashed on top of master and is running
> through Travis
>
> https://github.com/apache/flink/pull/9494
>
> Dyana
>
> On Tue, 20 Aug 2019 at 15:32, Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> > Hi Dyana,
> >
> > Regarding your question on the Chinese docs:
> > Since the Chinese counterparts for the Kinesis connector documentation
> > isn't translated yet (see docs/dev/connectors/kinesis.zh.md), for now
> you
> > can simply just sync whatever changes you made to the English doc to the
> > Chinese one as well.
> >
> > Cheers,
> > Gordon
> >
>

[Thomas Weise]

+1 for KCL 1.x changes only.

I also think it would make sense to align FLIP-27 work and KCL 2.x related
changes, since these will require a hardening cycle with extensive testing
that is probably not practical to repeat.



On Tue, Aug 20, 2019 at 10:57 AM Bowen Li <[hidden email]> wrote:

> @Stephan @Becket kinesis connector currently is using KCL 1.9. Mass changes
> are needed if we switch to KCL 2.x. I agree with Dyana that, since KCL 1.x
> is also updated to Apache 2.0, we can just focus on upgrading to a newer
> KCL 1.x minor version for now.
>
> On Tue, Aug 20, 2019 at 7:52 AM Dyana Rose <[hidden email]>
> wrote:
>
> > ok great,
> >
> > that's done, the PR is rebased and squashed on top of master and is
> running
> > through Travis
> >
> > https://github.com/apache/flink/pull/9494
> >
> > Dyana
> >
> > On Tue, 20 Aug 2019 at 15:32, Tzu-Li (Gordon) Tai <[hidden email]>
> > wrote:
> >
> > > Hi Dyana,
> > >
> > > Regarding your question on the Chinese docs:
> > > Since the Chinese counterparts for the Kinesis connector documentation
> > > isn't translated yet (see docs/dev/connectors/kinesis.zh.md), for now
> > you
> > > can simply just sync whatever changes you made to the English doc to
> the
> > > Chinese one as well.
> > >
> > > Cheers,
> > > Gordon
> > >
> >
>