Updating Kinesis Connector to latest Apache licensed libs

[Dyana Rose]

The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now on the Apache 2.0 license.
https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt

There is a PR for the Kinesis Producer Library to update it to the Apache 2.0 license (https://github.com/awslabs/amazon-kinesis-producer/pull/256)

If I understand the Amazon software license issue correctly updating to these new major versions (and the KPL when it's available under the Apache license) will allow the Kinesis connectors to be distributed in the core build. (making my life easier)

I haven't seen a Jira ticket specifically for an upgrade in major version, but it would solve this one, though otherwise than intended! https://issues.apache.org/jira/browse/FLINK-7673

Unless there are already discussed reasons not to upgrade, I'll stick a ticket in for it and cross my fingers that the KPL PR gets merged sometime in the relatively near future.

Thanks,
Dyana

[bowen.li]

Hi Dyana,

Thanks for bringing this up!

You are right that ASL is the blocker for us to officially include
flink-connecotr-kinesis as a connector module to build and publish to Maven
central. I've been thru the mess of building, publishing, and maintaining
flink-connector-kinesis via Jfrog, and that's a really really painful
experience... Glad to hear AWS finally pulls the trigger to change
KCL/KPL's license. So big +1 on this initiative from me.

I'm not aware of any previous discussion on this, so please feel free to
create a new JIRA ticket, assign to yourself, and work on it. As a
committer, I'll be happy to help move this effort forward, and we can seek
help from other experts in kinesis connector like @Tzu-Li (Gordon) Tai
<[hidden email]> when needed.

The task should include, but not limited to, upgrading KCL/KPL to new
versions of Apache 2.0 license, changing licenses and NOTICE files in
flink-connector-kinesis, and adding flink-connector-kinesis to build, CI
and artifact publishing pipeline. These can be broken into subtasks.

If AWS's PR you gave can be finished soon enough, we may be able to sneak
this into Flink 1.9 before feature freeze which is currently set as end of
June. Otherwise, we may have to wait till the next major release like 1.10,
as such a big change may not happen in maintenance releases like 1.9.1.

Bowen

On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <[hidden email]> wrote:

>
> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now on
> the Apache 2.0 license.
> https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
>
> There is a PR for the Kinesis Producer Library to update it to the Apache
> 2.0 license (https://github.com/awslabs/amazon-kinesis-producer/pull/256)
>
> If I understand the Amazon software license issue correctly updating to
> these new major versions (and the KPL when it's available under the Apache
> license) will allow the Kinesis connectors to be distributed in the core
> build. (making my life easier)
>
> I haven't seen a Jira ticket specifically for an upgrade in major version,
> but it would solve this one, though otherwise than intended!
> https://issues.apache.org/jira/browse/FLINK-7673
>
> Unless there are already discussed reasons not to upgrade, I'll stick a
> ticket in for it and cross my fingers that the KPL PR gets merged sometime
> in the relatively near future.
>
> Thanks,
> Dyana
>

[Tzu-Li (Gordon) Tai]

Hi,

Thanks Dyana for bringing this up and Bowen for helping to move this
forward. Very happy to hear about this!

Please feel free to to create a new JIRA ticket for this and assign it to
yourself. +1 to aim this for 1.9.0.
The subtasks that Bowen mentioned looks good to me. Issues such as [1] and
[2] should also be resolved as part of this effort.

Also cc'ing Thomas, who recently has more coverage on the Flink Kinesis
connector.

Cheers,
Gordon

[1]  https://issues.apache.org/jira/browse/FLINK-3924
[2]  https://issues.apache.org/jira/browse/FLINK-7673

On Fri, Jun 14, 2019 at 2:46 AM Bowen Li <[hidden email]> wrote:

> Hi Dyana,
>
> Thanks for bringing this up!
>
> You are right that ASL is the blocker for us to officially include
> flink-connecotr-kinesis as a connector module to build and publish to Maven
> central. I've been thru the mess of building, publishing, and maintaining
> flink-connector-kinesis via Jfrog, and that's a really really painful
> experience... Glad to hear AWS finally pulls the trigger to change
> KCL/KPL's license. So big +1 on this initiative from me.
>
> I'm not aware of any previous discussion on this, so please feel free to
> create a new JIRA ticket, assign to yourself, and work on it. As a
> committer, I'll be happy to help move this effort forward, and we can seek
> help from other experts in kinesis connector like @Tzu-Li (Gordon) Tai
> <[hidden email]> when needed.
>
> The task should include, but not limited to, upgrading KCL/KPL to new
> versions of Apache 2.0 license, changing licenses and NOTICE files in
> flink-connector-kinesis, and adding flink-connector-kinesis to build, CI
> and artifact publishing pipeline. These can be broken into subtasks.
>
> If AWS's PR you gave can be finished soon enough, we may be able to sneak
> this into Flink 1.9 before feature freeze which is currently set as end of
> June. Otherwise, we may have to wait till the next major release like 1.10,
> as such a big change may not happen in maintenance releases like 1.9.1.
>
> Bowen
>
> On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <[hidden email]>
> wrote:
>
>>
>> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now on
>> the Apache 2.0 license.
>> https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
>> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
>>
>> There is a PR for the Kinesis Producer Library to update it to the Apache
>> 2.0 license (https://github.com/awslabs/amazon-kinesis-producer/pull/256)
>>
>> If I understand the Amazon software license issue correctly updating to
>> these new major versions (and the KPL when it's available under the Apache
>> license) will allow the Kinesis connectors to be distributed in the core
>> build. (making my life easier)
>>
>> I haven't seen a Jira ticket specifically for an upgrade in major
>> version, but it would solve this one, though otherwise than intended!
>> https://issues.apache.org/jira/browse/FLINK-7673
>>
>> Unless there are already discussed reasons not to upgrade, I'll stick a
>> ticket in for it and cross my fingers that the KPL PR gets merged sometime
>> in the relatively near future.
>>
>> Thanks,
>> Dyana
>>
>

[Dyana Rose]

Brilliant. That Issue is in now under:
https://issues.apache.org/jira/browse/FLINK-12847

Thanks,
Dyana

On Fri, 14 Jun 2019 at 03:07, Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> Hi,
>
> Thanks Dyana for bringing this up and Bowen for helping to move this
> forward. Very happy to hear about this!
>
> Please feel free to to create a new JIRA ticket for this and assign it to
> yourself. +1 to aim this for 1.9.0.
> The subtasks that Bowen mentioned looks good to me. Issues such as [1] and
> [2] should also be resolved as part of this effort.
>
> Also cc'ing Thomas, who recently has more coverage on the Flink Kinesis
> connector.
>
> Cheers,
> Gordon
>
> [1]  https://issues.apache.org/jira/browse/FLINK-3924
> [2]  https://issues.apache.org/jira/browse/FLINK-7673
>
> On Fri, Jun 14, 2019 at 2:46 AM Bowen Li <[hidden email]> wrote:
>
> > Hi Dyana,
> >
> > Thanks for bringing this up!
> >
> > You are right that ASL is the blocker for us to officially include
> > flink-connecotr-kinesis as a connector module to build and publish to
> Maven
> > central. I've been thru the mess of building, publishing, and maintaining
> > flink-connector-kinesis via Jfrog, and that's a really really painful
> > experience... Glad to hear AWS finally pulls the trigger to change
> > KCL/KPL's license. So big +1 on this initiative from me.
> >
> > I'm not aware of any previous discussion on this, so please feel free to
> > create a new JIRA ticket, assign to yourself, and work on it. As a
> > committer, I'll be happy to help move this effort forward, and we can
> seek
> > help from other experts in kinesis connector like @Tzu-Li (Gordon) Tai
> > <[hidden email]> when needed.
> >
> > The task should include, but not limited to, upgrading KCL/KPL to new
> > versions of Apache 2.0 license, changing licenses and NOTICE files in
> > flink-connector-kinesis, and adding flink-connector-kinesis to build, CI
> > and artifact publishing pipeline. These can be broken into subtasks.
> >
> > If AWS's PR you gave can be finished soon enough, we may be able to sneak
> > this into Flink 1.9 before feature freeze which is currently set as end
> of
> > June. Otherwise, we may have to wait till the next major release like
> 1.10,
> > as such a big change may not happen in maintenance releases like 1.9.1.
> >
> > Bowen
> >
> > On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <[hidden email]>
> > wrote:
> >
> >>
> >> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now
> on
> >> the Apache 2.0 license.
> >>
> https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
> >> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
> >>
> >> There is a PR for the Kinesis Producer Library to update it to the
> Apache
> >> 2.0 license (
> https://github.com/awslabs/amazon-kinesis-producer/pull/256)
> >>
> >> If I understand the Amazon software license issue correctly updating to
> >> these new major versions (and the KPL when it's available under the
> Apache
> >> license) will allow the Kinesis connectors to be distributed in the core
> >> build. (making my life easier)
> >>
> >> I haven't seen a Jira ticket specifically for an upgrade in major
> >> version, but it would solve this one, though otherwise than intended!
> >> https://issues.apache.org/jira/browse/FLINK-7673
> >>
> >> Unless there are already discussed reasons not to upgrade, I'll stick a
> >> ticket in for it and cross my fingers that the KPL PR gets merged
> sometime
> >> in the relatively near future.
> >>
> >> Thanks,
> >> Dyana
> >>
> >
>

--

Dyana Rose
Software Engineer


W: www.salecycle.com <http://www.salecycle.com/>
[image: The 2019 Look Book - Download Now]
<https://t.xink.io/Tracking/Index/WcwBAKNtAAAwphkA0>

[Thomas Weise]

Dyana, thanks for taking this up!

The flink-connector-kinesis module is already part of the CI pipeline, it
is just excluded when creating the release. So what needs to be done is to
remove the -Pinclude-kinesis cruft and make it part of the default modules
instead.

Thomas


On Fri, Jun 14, 2019 at 10:06 AM Dyana Rose <[hidden email]>
wrote:

> Brilliant. That Issue is in now under:
> https://issues.apache.org/jira/browse/FLINK-12847
>
> Thanks,
> Dyana
>
> On Fri, 14 Jun 2019 at 03:07, Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Thanks Dyana for bringing this up and Bowen for helping to move this
> > forward. Very happy to hear about this!
> >
> > Please feel free to to create a new JIRA ticket for this and assign it to
> > yourself. +1 to aim this for 1.9.0.
> > The subtasks that Bowen mentioned looks good to me. Issues such as [1]
> and
> > [2] should also be resolved as part of this effort.
> >
> > Also cc'ing Thomas, who recently has more coverage on the Flink Kinesis
> > connector.
> >
> > Cheers,
> > Gordon
> >
> > [1]  https://issues.apache.org/jira/browse/FLINK-3924
> > [2]  https://issues.apache.org/jira/browse/FLINK-7673
> >
> > On Fri, Jun 14, 2019 at 2:46 AM Bowen Li <[hidden email]> wrote:
> >
> > > Hi Dyana,
> > >
> > > Thanks for bringing this up!
> > >
> > > You are right that ASL is the blocker for us to officially include
> > > flink-connecotr-kinesis as a connector module to build and publish to
> > Maven
> > > central. I've been thru the mess of building, publishing, and
> maintaining
> > > flink-connector-kinesis via Jfrog, and that's a really really painful
> > > experience... Glad to hear AWS finally pulls the trigger to change
> > > KCL/KPL's license. So big +1 on this initiative from me.
> > >
> > > I'm not aware of any previous discussion on this, so please feel free
> to
> > > create a new JIRA ticket, assign to yourself, and work on it. As a
> > > committer, I'll be happy to help move this effort forward, and we can
> > seek
> > > help from other experts in kinesis connector like @Tzu-Li (Gordon) Tai
> > > <[hidden email]> when needed.
> > >
> > > The task should include, but not limited to, upgrading KCL/KPL to new
> > > versions of Apache 2.0 license, changing licenses and NOTICE files in
> > > flink-connector-kinesis, and adding flink-connector-kinesis to build,
> CI
> > > and artifact publishing pipeline. These can be broken into subtasks.
> > >
> > > If AWS's PR you gave can be finished soon enough, we may be able to
> sneak
> > > this into Flink 1.9 before feature freeze which is currently set as end
> > of
> > > June. Otherwise, we may have to wait till the next major release like
> > 1.10,
> > > as such a big change may not happen in maintenance releases like 1.9.1.
> > >
> > > Bowen
> > >
> > > On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <[hidden email]>
> > > wrote:
> > >
> > >>
> > >> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now
> > on
> > >> the Apache 2.0 license.
> > >>
> > https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
> > >> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
> > >>
> > >> There is a PR for the Kinesis Producer Library to update it to the
> > Apache
> > >> 2.0 license (
> > https://github.com/awslabs/amazon-kinesis-producer/pull/256)
> > >>
> > >> If I understand the Amazon software license issue correctly updating
> to
> > >> these new major versions (and the KPL when it's available under the
> > Apache
> > >> license) will allow the Kinesis connectors to be distributed in the
> core
> > >> build. (making my life easier)
> > >>
> > >> I haven't seen a Jira ticket specifically for an upgrade in major
> > >> version, but it would solve this one, though otherwise than intended!
> > >> https://issues.apache.org/jira/browse/FLINK-7673
> > >>
> > >> Unless there are already discussed reasons not to upgrade, I'll stick
> a
> > >> ticket in for it and cross my fingers that the KPL PR gets merged
> > sometime
> > >> in the relatively near future.
> > >>
> > >> Thanks,
> > >> Dyana
> > >>
> > >
> >
>
>
> --
>
> Dyana Rose
> Software Engineer
>
>
> W: www.salecycle.com <http://www.salecycle.com/>
> [image: The 2019 Look Book - Download Now]
> <https://t.xink.io/Tracking/Index/WcwBAKNtAAAwphkA0>
>

[Aljoscha Krettek-2]

+1

Nice! Less special-case handling is always good.

> On 14. Jun 2019, at 10:30, Thomas Weise <[hidden email]> wrote:
>
> Dyana, thanks for taking this up!
>
> The flink-connector-kinesis module is already part of the CI pipeline, it
> is just excluded when creating the release. So what needs to be done is to
> remove the -Pinclude-kinesis cruft and make it part of the default modules
> instead.
>
> Thomas
>
>
> On Fri, Jun 14, 2019 at 10:06 AM Dyana Rose <[hidden email]>
> wrote:
>
>> Brilliant. That Issue is in now under:
>> https://issues.apache.org/jira/browse/FLINK-12847
>>
>> Thanks,
>> Dyana
>>
>> On Fri, 14 Jun 2019 at 03:07, Tzu-Li (Gordon) Tai <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks Dyana for bringing this up and Bowen for helping to move this
>>> forward. Very happy to hear about this!
>>>
>>> Please feel free to to create a new JIRA ticket for this and assign it to
>>> yourself. +1 to aim this for 1.9.0.
>>> The subtasks that Bowen mentioned looks good to me. Issues such as [1]
>> and
>>> [2] should also be resolved as part of this effort.
>>>
>>> Also cc'ing Thomas, who recently has more coverage on the Flink Kinesis
>>> connector.
>>>
>>> Cheers,
>>> Gordon
>>>
>>> [1]  https://issues.apache.org/jira/browse/FLINK-3924
>>> [2]  https://issues.apache.org/jira/browse/FLINK-7673
>>>
>>> On Fri, Jun 14, 2019 at 2:46 AM Bowen Li <[hidden email]> wrote:
>>>
>>>> Hi Dyana,
>>>>
>>>> Thanks for bringing this up!
>>>>
>>>> You are right that ASL is the blocker for us to officially include
>>>> flink-connecotr-kinesis as a connector module to build and publish to
>>> Maven
>>>> central. I've been thru the mess of building, publishing, and
>> maintaining
>>>> flink-connector-kinesis via Jfrog, and that's a really really painful
>>>> experience... Glad to hear AWS finally pulls the trigger to change
>>>> KCL/KPL's license. So big +1 on this initiative from me.
>>>>
>>>> I'm not aware of any previous discussion on this, so please feel free
>> to
>>>> create a new JIRA ticket, assign to yourself, and work on it. As a
>>>> committer, I'll be happy to help move this effort forward, and we can
>>> seek
>>>> help from other experts in kinesis connector like @Tzu-Li (Gordon) Tai
>>>> <[hidden email]> when needed.
>>>>
>>>> The task should include, but not limited to, upgrading KCL/KPL to new
>>>> versions of Apache 2.0 license, changing licenses and NOTICE files in
>>>> flink-connector-kinesis, and adding flink-connector-kinesis to build,
>> CI
>>>> and artifact publishing pipeline. These can be broken into subtasks.
>>>>
>>>> If AWS's PR you gave can be finished soon enough, we may be able to
>> sneak
>>>> this into Flink 1.9 before feature freeze which is currently set as end
>>> of
>>>> June. Otherwise, we may have to wait till the next major release like
>>> 1.10,
>>>> as such a big change may not happen in maintenance releases like 1.9.1.
>>>>
>>>> Bowen
>>>>
>>>> On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <[hidden email]>
>>>> wrote:
>>>>
>>>>>
>>>>> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are now
>>> on
>>>>> the Apache 2.0 license.
>>>>>
>>> https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
>>>>> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
>>>>>
>>>>> There is a PR for the Kinesis Producer Library to update it to the
>>> Apache
>>>>> 2.0 license (
>>> https://github.com/awslabs/amazon-kinesis-producer/pull/256)
>>>>>
>>>>> If I understand the Amazon software license issue correctly updating
>> to
>>>>> these new major versions (and the KPL when it's available under the
>>> Apache
>>>>> license) will allow the Kinesis connectors to be distributed in the
>> core
>>>>> build. (making my life easier)
>>>>>
>>>>> I haven't seen a Jira ticket specifically for an upgrade in major
>>>>> version, but it would solve this one, though otherwise than intended!
>>>>> https://issues.apache.org/jira/browse/FLINK-7673
>>>>>
>>>>> Unless there are already discussed reasons not to upgrade, I'll stick
>> a
>>>>> ticket in for it and cross my fingers that the KPL PR gets merged
>>> sometime
>>>>> in the relatively near future.
>>>>>
>>>>> Thanks,
>>>>> Dyana
>>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Dyana Rose
>> Software Engineer
>>
>>
>> W: www.salecycle.com <http://www.salecycle.com/>
>> [image: The 2019 Look Book - Download Now]
>> <https://t.xink.io/Tracking/Index/WcwBAKNtAAAwphkA0>
>>

[Dyana Rose]

I've pushed an early WIP of the code changes to our fork and added a long
comment on the work that's been done, what issues I've come across, and
requests for discussion on those issues

https://issues.apache.org/jira/browse/FLINK-12847

Thanks,
Dyana

On Fri, 14 Jun 2019 at 10:34, Aljoscha Krettek <[hidden email]> wrote:

> +1
>
> Nice! Less special-case handling is always good.
>
> > On 14. Jun 2019, at 10:30, Thomas Weise <[hidden email]> wrote:
> >
> > Dyana, thanks for taking this up!
> >
> > The flink-connector-kinesis module is already part of the CI pipeline, it
> > is just excluded when creating the release. So what needs to be done is
> to
> > remove the -Pinclude-kinesis cruft and make it part of the default
> modules
> > instead.
> >
> > Thomas
> >
> >
> > On Fri, Jun 14, 2019 at 10:06 AM Dyana Rose <[hidden email]>
> > wrote:
> >
> >> Brilliant. That Issue is in now under:
> >> https://issues.apache.org/jira/browse/FLINK-12847
> >>
> >> Thanks,
> >> Dyana
> >>
> >> On Fri, 14 Jun 2019 at 03:07, Tzu-Li (Gordon) Tai <[hidden email]>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks Dyana for bringing this up and Bowen for helping to move this
> >>> forward. Very happy to hear about this!
> >>>
> >>> Please feel free to to create a new JIRA ticket for this and assign it
> to
> >>> yourself. +1 to aim this for 1.9.0.
> >>> The subtasks that Bowen mentioned looks good to me. Issues such as [1]
> >> and
> >>> [2] should also be resolved as part of this effort.
> >>>
> >>> Also cc'ing Thomas, who recently has more coverage on the Flink Kinesis
> >>> connector.
> >>>
> >>> Cheers,
> >>> Gordon
> >>>
> >>> [1]  https://issues.apache.org/jira/browse/FLINK-3924
> >>> [2]  https://issues.apache.org/jira/browse/FLINK-7673
> >>>
> >>> On Fri, Jun 14, 2019 at 2:46 AM Bowen Li <[hidden email]> wrote:
> >>>
> >>>> Hi Dyana,
> >>>>
> >>>> Thanks for bringing this up!
> >>>>
> >>>> You are right that ASL is the blocker for us to officially include
> >>>> flink-connecotr-kinesis as a connector module to build and publish to
> >>> Maven
> >>>> central. I've been thru the mess of building, publishing, and
> >> maintaining
> >>>> flink-connector-kinesis via Jfrog, and that's a really really painful
> >>>> experience... Glad to hear AWS finally pulls the trigger to change
> >>>> KCL/KPL's license. So big +1 on this initiative from me.
> >>>>
> >>>> I'm not aware of any previous discussion on this, so please feel free
> >> to
> >>>> create a new JIRA ticket, assign to yourself, and work on it. As a
> >>>> committer, I'll be happy to help move this effort forward, and we can
> >>> seek
> >>>> help from other experts in kinesis connector like @Tzu-Li (Gordon) Tai
> >>>> <[hidden email]> when needed.
> >>>>
> >>>> The task should include, but not limited to, upgrading KCL/KPL to new
> >>>> versions of Apache 2.0 license, changing licenses and NOTICE files in
> >>>> flink-connector-kinesis, and adding flink-connector-kinesis to build,
> >> CI
> >>>> and artifact publishing pipeline. These can be broken into subtasks.
> >>>>
> >>>> If AWS's PR you gave can be finished soon enough, we may be able to
> >> sneak
> >>>> this into Flink 1.9 before feature freeze which is currently set as
> end
> >>> of
> >>>> June. Otherwise, we may have to wait till the next major release like
> >>> 1.10,
> >>>> as such a big change may not happen in maintenance releases like
> 1.9.1.
> >>>>
> >>>> Bowen
> >>>>
> >>>> On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>>
> >>>>> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are
> now
> >>> on
> >>>>> the Apache 2.0 license.
> >>>>>
> >>>
> https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
> >>>>> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
> >>>>>
> >>>>> There is a PR for the Kinesis Producer Library to update it to the
> >>> Apache
> >>>>> 2.0 license (
> >>> https://github.com/awslabs/amazon-kinesis-producer/pull/256)
> >>>>>
> >>>>> If I understand the Amazon software license issue correctly updating
> >> to
> >>>>> these new major versions (and the KPL when it's available under the
> >>> Apache
> >>>>> license) will allow the Kinesis connectors to be distributed in the
> >> core
> >>>>> build. (making my life easier)
> >>>>>
> >>>>> I haven't seen a Jira ticket specifically for an upgrade in major
> >>>>> version, but it would solve this one, though otherwise than intended!
> >>>>> https://issues.apache.org/jira/browse/FLINK-7673
> >>>>>
> >>>>> Unless there are already discussed reasons not to upgrade, I'll stick
> >> a
> >>>>> ticket in for it and cross my fingers that the KPL PR gets merged
> >>> sometime
> >>>>> in the relatively near future.
> >>>>>
> >>>>> Thanks,
> >>>>> Dyana
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >>
> >> Dyana Rose
> >> Software Engineer
> >>
> >>
> >> W: www.salecycle.com <http://www.salecycle.com/>
> >> [image: The 2019 Look Book - Download Now]
> >> <https://t.xink.io/Tracking/Index/WcwBAKNtAAAwphkA0>
> >>
>
>

--

Dyana Rose
Software Engineer


W: www.salecycle.com <http://www.salecycle.com/>
[image: Airline & Travel Booking Trends - Download Report]
<https://t.xink.io/Tracking/Index/vM4BAKNtAAAwphkA0>

[bowen.li]

Hi all,

I want to give an update from my perspective.

flink-connector-kinesis' two critical dependencies KPL and Dynamo stream
adapter haven't been converted to Apache 2.0 license yet. The
license-changing PRs [1] are still pending, for ~2 months so far. I've left
comments there, and hope they can be merged and have new releases soon.

Given that, as well as some major work required to adapt
flink-connector-kinesis code to new AWS libs (described by Dyana in the
ticket), I personally don't think we can make it in 1.9.

I would recommend re-aiming it at 1.10.

[1] KPL https://github.com/awslabs/amazon-kinesis-producer/pull/256
     Dynamo stream adapter
https://github.com/awslabs/dynamodb-streams-kinesis-adapter/pull/23

On Tue, Jun 18, 2019 at 6:09 AM Dyana Rose <[hidden email]> wrote:

> I've pushed an early WIP of the code changes to our fork and added a long
> comment on the work that's been done, what issues I've come across, and
> requests for discussion on those issues
>
> https://issues.apache.org/jira/browse/FLINK-12847
>
> Thanks,
> Dyana
>
> On Fri, 14 Jun 2019 at 10:34, Aljoscha Krettek <[hidden email]>
> wrote:
>
> > +1
> >
> > Nice! Less special-case handling is always good.
> >
> > > On 14. Jun 2019, at 10:30, Thomas Weise <[hidden email]> wrote:
> > >
> > > Dyana, thanks for taking this up!
> > >
> > > The flink-connector-kinesis module is already part of the CI pipeline,
> it
> > > is just excluded when creating the release. So what needs to be done is
> > to
> > > remove the -Pinclude-kinesis cruft and make it part of the default
> > modules
> > > instead.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Jun 14, 2019 at 10:06 AM Dyana Rose <[hidden email]>
> > > wrote:
> > >
> > >> Brilliant. That Issue is in now under:
> > >> https://issues.apache.org/jira/browse/FLINK-12847
> > >>
> > >> Thanks,
> > >> Dyana
> > >>
> > >> On Fri, 14 Jun 2019 at 03:07, Tzu-Li (Gordon) Tai <
> [hidden email]>
> > >> wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> Thanks Dyana for bringing this up and Bowen for helping to move this
> > >>> forward. Very happy to hear about this!
> > >>>
> > >>> Please feel free to to create a new JIRA ticket for this and assign
> it
> > to
> > >>> yourself. +1 to aim this for 1.9.0.
> > >>> The subtasks that Bowen mentioned looks good to me. Issues such as
> [1]
> > >> and
> > >>> [2] should also be resolved as part of this effort.
> > >>>
> > >>> Also cc'ing Thomas, who recently has more coverage on the Flink
> Kinesis
> > >>> connector.
> > >>>
> > >>> Cheers,
> > >>> Gordon
> > >>>
> > >>> [1]  https://issues.apache.org/jira/browse/FLINK-3924
> > >>> [2]  https://issues.apache.org/jira/browse/FLINK-7673
> > >>>
> > >>> On Fri, Jun 14, 2019 at 2:46 AM Bowen Li <[hidden email]>
> wrote:
> > >>>
> > >>>> Hi Dyana,
> > >>>>
> > >>>> Thanks for bringing this up!
> > >>>>
> > >>>> You are right that ASL is the blocker for us to officially include
> > >>>> flink-connecotr-kinesis as a connector module to build and publish
> to
> > >>> Maven
> > >>>> central. I've been thru the mess of building, publishing, and
> > >> maintaining
> > >>>> flink-connector-kinesis via Jfrog, and that's a really really
> painful
> > >>>> experience... Glad to hear AWS finally pulls the trigger to change
> > >>>> KCL/KPL's license. So big +1 on this initiative from me.
> > >>>>
> > >>>> I'm not aware of any previous discussion on this, so please feel
> free
> > >> to
> > >>>> create a new JIRA ticket, assign to yourself, and work on it. As a
> > >>>> committer, I'll be happy to help move this effort forward, and we
> can
> > >>> seek
> > >>>> help from other experts in kinesis connector like @Tzu-Li (Gordon)
> Tai
> > >>>> <[hidden email]> when needed.
> > >>>>
> > >>>> The task should include, but not limited to, upgrading KCL/KPL to
> new
> > >>>> versions of Apache 2.0 license, changing licenses and NOTICE files
> in
> > >>>> flink-connector-kinesis, and adding flink-connector-kinesis to
> build,
> > >> CI
> > >>>> and artifact publishing pipeline. These can be broken into subtasks.
> > >>>>
> > >>>> If AWS's PR you gave can be finished soon enough, we may be able to
> > >> sneak
> > >>>> this into Flink 1.9 before feature freeze which is currently set as
> > end
> > >>> of
> > >>>> June. Otherwise, we may have to wait till the next major release
> like
> > >>> 1.10,
> > >>>> as such a big change may not happen in maintenance releases like
> > 1.9.1.
> > >>>>
> > >>>> Bowen
> > >>>>
> > >>>> On Thu, Jun 13, 2019 at 5:38 AM dyana.rose <
> [hidden email]>
> > >>>> wrote:
> > >>>>
> > >>>>>
> > >>>>> The Kinesis Client Library v2.x and the AWS Java SDK v2.x both are
> > now
> > >>> on
> > >>>>> the Apache 2.0 license.
> > >>>>>
> > >>>
> > https://github.com/awslabs/amazon-kinesis-client/blob/master/LICENSE.txt
> > >>>>> https://github.com/aws/aws-sdk-java-v2/blob/master/LICENSE.txt
> > >>>>>
> > >>>>> There is a PR for the Kinesis Producer Library to update it to the
> > >>> Apache
> > >>>>> 2.0 license (
> > >>> https://github.com/awslabs/amazon-kinesis-producer/pull/256)
> > >>>>>
> > >>>>> If I understand the Amazon software license issue correctly
> updating
> > >> to
> > >>>>> these new major versions (and the KPL when it's available under the
> > >>> Apache
> > >>>>> license) will allow the Kinesis connectors to be distributed in the
> > >> core
> > >>>>> build. (making my life easier)
> > >>>>>
> > >>>>> I haven't seen a Jira ticket specifically for an upgrade in major
> > >>>>> version, but it would solve this one, though otherwise than
> intended!
> > >>>>> https://issues.apache.org/jira/browse/FLINK-7673
> > >>>>>
> > >>>>> Unless there are already discussed reasons not to upgrade, I'll
> stick
> > >> a
> > >>>>> ticket in for it and cross my fingers that the KPL PR gets merged
> > >>> sometime
> > >>>>> in the relatively near future.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Dyana
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Dyana Rose
> > >> Software Engineer
> > >>
> > >>
> > >> W: www.salecycle.com <http://www.salecycle.com/>
> > >> [image: The 2019 Look Book - Download Now]
> > >> <https://t.xink.io/Tracking/Index/WcwBAKNtAAAwphkA0>
> > >>
> >
> >
>
> --
>
> Dyana Rose
> Software Engineer
>
>
> W: www.salecycle.com <http://www.salecycle.com/>
> [image: Airline & Travel Booking Trends - Download Report]
> <https://t.xink.io/Tracking/Index/vM4BAKNtAAAwphkA0>
>