Dependency convergence

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Dependency convergence

Piotr Nowojski
Hi,

I have spent last couple of days trying to find and fix Kafka tests instabilities on Travis and I think I have finally found the main reason: dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper 3.10. Effect was very subtle, because rarely in some corner cases (but not always) Netty was deadlocking itself…

Because of that I would like to enable dependencyConvergence rule in maven-enforcer-plugin project wide - it catches this error immediately:

Dependency convergence error for io.netty:netty:3.10.5.Final paths to dependency are:
+-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
  +-org.apache.kafka:kafka_2.11:0.9.0.1
    +-org.apache.zookeeper:zookeeper:3.4.10
      +-io.netty:netty:3.10.5.Final
and
+-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
  +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
    +-com.data-artisans:flakka-remote_2.11:2.3-custom
      +-io.netty:netty:3.8.0.Final

Currently this rule fails with multiple errors, but after those lost couple of days I’m pretty determined to fix all of them “just in case”. dependencyConvergence rule would protect us in the future against such nasty subtle bugs. Does anyone have any objections/issues that I’m not aware of?

Piotrek
Reply | Threaded
Open this post in threaded view
|

Re: Dependency convergence

Bowen Li
+1. This is great, Piotrek!

BTW, can you clarify what you mean by 'project wide'? Is it the whole
`flink` project or just `flink-connector-kafka`? I think it's useful to
be applied to the whole flink project. I've seen dependencies conflict
problem like this in flink-connector-kinesis. Enabling this in flink would
protect us from many hidden issues.

Bowen



On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]>
wrote:

> Hi,
>
> I have spent last couple of days trying to find and fix Kafka tests
> instabilities on Travis and I think I have finally found the main reason:
> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper 3.10.
> Effect was very subtle, because rarely in some corner cases (but not
> always) Netty was deadlocking itself…
>
> Because of that I would like to enable dependencyConvergence rule in
> maven-enforcer-plugin project wide - it catches this error immediately:
>
> Dependency convergence error for io.netty:netty:3.10.5.Final paths to
> dependency are:
> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
>   +-org.apache.kafka:kafka_2.11:0.9.0.1
>     +-org.apache.zookeeper:zookeeper:3.4.10
>       +-io.netty:netty:3.10.5.Final
> and
> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
>   +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
>     +-com.data-artisans:flakka-remote_2.11:2.3-custom
>       +-io.netty:netty:3.8.0.Final
>
> Currently this rule fails with multiple errors, but after those lost
> couple of days I’m pretty determined to fix all of them “just in case”.
> dependencyConvergence rule would protect us in the future against such
> nasty subtle bugs. Does anyone have any objections/issues that I’m not
> aware of?
>
> Piotrek
Reply | Threaded
Open this post in threaded view
|

Re: Dependency convergence

Piotr Nowojski
I meant for whole flink project.

> On Oct 4, 2017, at 8:43 PM, Bowen Li <[hidden email]> wrote:
>
> +1. This is great, Piotrek!
>
> BTW, can you clarify what you mean by 'project wide'? Is it the whole
> `flink` project or just `flink-connector-kafka`? I think it's useful to
> be applied to the whole flink project. I've seen dependencies conflict
> problem like this in flink-connector-kinesis. Enabling this in flink would
> protect us from many hidden issues.
>
> Bowen
>
>
>
> On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]>
> wrote:
>
>> Hi,
>>
>> I have spent last couple of days trying to find and fix Kafka tests
>> instabilities on Travis and I think I have finally found the main reason:
>> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper 3.10.
>> Effect was very subtle, because rarely in some corner cases (but not
>> always) Netty was deadlocking itself…
>>
>> Because of that I would like to enable dependencyConvergence rule in
>> maven-enforcer-plugin project wide - it catches this error immediately:
>>
>> Dependency convergence error for io.netty:netty:3.10.5.Final paths to
>> dependency are:
>> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
>>  +-org.apache.kafka:kafka_2.11:0.9.0.1
>>    +-org.apache.zookeeper:zookeeper:3.4.10
>>      +-io.netty:netty:3.10.5.Final
>> and
>> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
>>  +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
>>    +-com.data-artisans:flakka-remote_2.11:2.3-custom
>>      +-io.netty:netty:3.8.0.Final
>>
>> Currently this rule fails with multiple errors, but after those lost
>> couple of days I’m pretty determined to fix all of them “just in case”.
>> dependencyConvergence rule would protect us in the future against such
>> nasty subtle bugs. Does anyone have any objections/issues that I’m not
>> aware of?
>>
>> Piotrek

Reply | Threaded
Open this post in threaded view
|

Re: Dependency convergence

Till Rohrmann
+1 for pulling our dependencies straight and guarding it via the
maven-enforcer-plugin.

On Wed, Oct 4, 2017 at 8:58 PM, Piotr Nowojski <[hidden email]>
wrote:

> I meant for whole flink project.
>
> > On Oct 4, 2017, at 8:43 PM, Bowen Li <[hidden email]> wrote:
> >
> > +1. This is great, Piotrek!
> >
> > BTW, can you clarify what you mean by 'project wide'? Is it the whole
> > `flink` project or just `flink-connector-kafka`? I think it's useful to
> > be applied to the whole flink project. I've seen dependencies conflict
> > problem like this in flink-connector-kinesis. Enabling this in flink
> would
> > protect us from many hidden issues.
> >
> > Bowen
> >
> >
> >
> > On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]>
> > wrote:
> >
> >> Hi,
> >>
> >> I have spent last couple of days trying to find and fix Kafka tests
> >> instabilities on Travis and I think I have finally found the main
> reason:
> >> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper
> 3.10.
> >> Effect was very subtle, because rarely in some corner cases (but not
> >> always) Netty was deadlocking itself…
> >>
> >> Because of that I would like to enable dependencyConvergence rule in
> >> maven-enforcer-plugin project wide - it catches this error immediately:
> >>
> >> Dependency convergence error for io.netty:netty:3.10.5.Final paths to
> >> dependency are:
> >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
> >>  +-org.apache.kafka:kafka_2.11:0.9.0.1
> >>    +-org.apache.zookeeper:zookeeper:3.4.10
> >>      +-io.netty:netty:3.10.5.Final
> >> and
> >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
> >>  +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
> >>    +-com.data-artisans:flakka-remote_2.11:2.3-custom
> >>      +-io.netty:netty:3.8.0.Final
> >>
> >> Currently this rule fails with multiple errors, but after those lost
> >> couple of days I’m pretty determined to fix all of them “just in case”.
> >> dependencyConvergence rule would protect us in the future against such
> >> nasty subtle bugs. Does anyone have any objections/issues that I’m not
> >> aware of?
> >>
> >> Piotrek
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Dependency convergence

Aljoscha Krettek-2
+2 ;-)

> On 5. Oct 2017, at 11:26, Till Rohrmann <[hidden email]> wrote:
>
> +1 for pulling our dependencies straight and guarding it via the
> maven-enforcer-plugin.
>
> On Wed, Oct 4, 2017 at 8:58 PM, Piotr Nowojski <[hidden email]>
> wrote:
>
>> I meant for whole flink project.
>>
>>> On Oct 4, 2017, at 8:43 PM, Bowen Li <[hidden email]> wrote:
>>>
>>> +1. This is great, Piotrek!
>>>
>>> BTW, can you clarify what you mean by 'project wide'? Is it the whole
>>> `flink` project or just `flink-connector-kafka`? I think it's useful to
>>> be applied to the whole flink project. I've seen dependencies conflict
>>> problem like this in flink-connector-kinesis. Enabling this in flink
>> would
>>> protect us from many hidden issues.
>>>
>>> Bowen
>>>
>>>
>>>
>>> On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have spent last couple of days trying to find and fix Kafka tests
>>>> instabilities on Travis and I think I have finally found the main
>> reason:
>>>> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper
>> 3.10.
>>>> Effect was very subtle, because rarely in some corner cases (but not
>>>> always) Netty was deadlocking itself…
>>>>
>>>> Because of that I would like to enable dependencyConvergence rule in
>>>> maven-enforcer-plugin project wide - it catches this error immediately:
>>>>
>>>> Dependency convergence error for io.netty:netty:3.10.5.Final paths to
>>>> dependency are:
>>>> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
>>>> +-org.apache.kafka:kafka_2.11:0.9.0.1
>>>>   +-org.apache.zookeeper:zookeeper:3.4.10
>>>>     +-io.netty:netty:3.10.5.Final
>>>> and
>>>> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT
>>>> +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT
>>>>   +-com.data-artisans:flakka-remote_2.11:2.3-custom
>>>>     +-io.netty:netty:3.8.0.Final
>>>>
>>>> Currently this rule fails with multiple errors, but after those lost
>>>> couple of days I’m pretty determined to fix all of them “just in case”.
>>>> dependencyConvergence rule would protect us in the future against such
>>>> nasty subtle bugs. Does anyone have any objections/issues that I’m not
>>>> aware of?
>>>>
>>>> Piotrek
>>
>>