Hi,
I have spent last couple of days trying to find and fix Kafka tests instabilities on Travis and I think I have finally found the main reason: dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper 3.10. Effect was very subtle, because rarely in some corner cases (but not always) Netty was deadlocking itself… Because of that I would like to enable dependencyConvergence rule in maven-enforcer-plugin project wide - it catches this error immediately: Dependency convergence error for io.netty:netty:3.10.5.Final paths to dependency are: +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT +-org.apache.kafka:kafka_2.11:0.9.0.1 +-org.apache.zookeeper:zookeeper:3.4.10 +-io.netty:netty:3.10.5.Final and +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT +-com.data-artisans:flakka-remote_2.11:2.3-custom +-io.netty:netty:3.8.0.Final Currently this rule fails with multiple errors, but after those lost couple of days I’m pretty determined to fix all of them “just in case”. dependencyConvergence rule would protect us in the future against such nasty subtle bugs. Does anyone have any objections/issues that I’m not aware of? Piotrek |
+1. This is great, Piotrek!
BTW, can you clarify what you mean by 'project wide'? Is it the whole `flink` project or just `flink-connector-kafka`? I think it's useful to be applied to the whole flink project. I've seen dependencies conflict problem like this in flink-connector-kinesis. Enabling this in flink would protect us from many hidden issues. Bowen On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]> wrote: > Hi, > > I have spent last couple of days trying to find and fix Kafka tests > instabilities on Travis and I think I have finally found the main reason: > dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper 3.10. > Effect was very subtle, because rarely in some corner cases (but not > always) Netty was deadlocking itself… > > Because of that I would like to enable dependencyConvergence rule in > maven-enforcer-plugin project wide - it catches this error immediately: > > Dependency convergence error for io.netty:netty:3.10.5.Final paths to > dependency are: > +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT > +-org.apache.kafka:kafka_2.11:0.9.0.1 > +-org.apache.zookeeper:zookeeper:3.4.10 > +-io.netty:netty:3.10.5.Final > and > +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT > +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT > +-com.data-artisans:flakka-remote_2.11:2.3-custom > +-io.netty:netty:3.8.0.Final > > Currently this rule fails with multiple errors, but after those lost > couple of days I’m pretty determined to fix all of them “just in case”. > dependencyConvergence rule would protect us in the future against such > nasty subtle bugs. Does anyone have any objections/issues that I’m not > aware of? > > Piotrek |
I meant for whole flink project.
> On Oct 4, 2017, at 8:43 PM, Bowen Li <[hidden email]> wrote: > > +1. This is great, Piotrek! > > BTW, can you clarify what you mean by 'project wide'? Is it the whole > `flink` project or just `flink-connector-kafka`? I think it's useful to > be applied to the whole flink project. I've seen dependencies conflict > problem like this in flink-connector-kinesis. Enabling this in flink would > protect us from many hidden issues. > > Bowen > > > > On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]> > wrote: > >> Hi, >> >> I have spent last couple of days trying to find and fix Kafka tests >> instabilities on Travis and I think I have finally found the main reason: >> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper 3.10. >> Effect was very subtle, because rarely in some corner cases (but not >> always) Netty was deadlocking itself… >> >> Because of that I would like to enable dependencyConvergence rule in >> maven-enforcer-plugin project wide - it catches this error immediately: >> >> Dependency convergence error for io.netty:netty:3.10.5.Final paths to >> dependency are: >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT >> +-org.apache.kafka:kafka_2.11:0.9.0.1 >> +-org.apache.zookeeper:zookeeper:3.4.10 >> +-io.netty:netty:3.10.5.Final >> and >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT >> +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT >> +-com.data-artisans:flakka-remote_2.11:2.3-custom >> +-io.netty:netty:3.8.0.Final >> >> Currently this rule fails with multiple errors, but after those lost >> couple of days I’m pretty determined to fix all of them “just in case”. >> dependencyConvergence rule would protect us in the future against such >> nasty subtle bugs. Does anyone have any objections/issues that I’m not >> aware of? >> >> Piotrek |
+1 for pulling our dependencies straight and guarding it via the
maven-enforcer-plugin. On Wed, Oct 4, 2017 at 8:58 PM, Piotr Nowojski <[hidden email]> wrote: > I meant for whole flink project. > > > On Oct 4, 2017, at 8:43 PM, Bowen Li <[hidden email]> wrote: > > > > +1. This is great, Piotrek! > > > > BTW, can you clarify what you mean by 'project wide'? Is it the whole > > `flink` project or just `flink-connector-kafka`? I think it's useful to > > be applied to the whole flink project. I've seen dependencies conflict > > problem like this in flink-connector-kinesis. Enabling this in flink > would > > protect us from many hidden issues. > > > > Bowen > > > > > > > > On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]> > > wrote: > > > >> Hi, > >> > >> I have spent last couple of days trying to find and fix Kafka tests > >> instabilities on Travis and I think I have finally found the main > reason: > >> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper > 3.10. > >> Effect was very subtle, because rarely in some corner cases (but not > >> always) Netty was deadlocking itself… > >> > >> Because of that I would like to enable dependencyConvergence rule in > >> maven-enforcer-plugin project wide - it catches this error immediately: > >> > >> Dependency convergence error for io.netty:netty:3.10.5.Final paths to > >> dependency are: > >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT > >> +-org.apache.kafka:kafka_2.11:0.9.0.1 > >> +-org.apache.zookeeper:zookeeper:3.4.10 > >> +-io.netty:netty:3.10.5.Final > >> and > >> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT > >> +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT > >> +-com.data-artisans:flakka-remote_2.11:2.3-custom > >> +-io.netty:netty:3.8.0.Final > >> > >> Currently this rule fails with multiple errors, but after those lost > >> couple of days I’m pretty determined to fix all of them “just in case”. > >> dependencyConvergence rule would protect us in the future against such > >> nasty subtle bugs. Does anyone have any objections/issues that I’m not > >> aware of? > >> > >> Piotrek > > |
+2 ;-)
> On 5. Oct 2017, at 11:26, Till Rohrmann <[hidden email]> wrote: > > +1 for pulling our dependencies straight and guarding it via the > maven-enforcer-plugin. > > On Wed, Oct 4, 2017 at 8:58 PM, Piotr Nowojski <[hidden email]> > wrote: > >> I meant for whole flink project. >> >>> On Oct 4, 2017, at 8:43 PM, Bowen Li <[hidden email]> wrote: >>> >>> +1. This is great, Piotrek! >>> >>> BTW, can you clarify what you mean by 'project wide'? Is it the whole >>> `flink` project or just `flink-connector-kafka`? I think it's useful to >>> be applied to the whole flink project. I've seen dependencies conflict >>> problem like this in flink-connector-kinesis. Enabling this in flink >> would >>> protect us from many hidden issues. >>> >>> Bowen >>> >>> >>> >>> On Wed, Oct 4, 2017 at 9:39 AM, Piotr Nowojski <[hidden email]> >>> wrote: >>> >>>> Hi, >>>> >>>> I have spent last couple of days trying to find and fix Kafka tests >>>> instabilities on Travis and I think I have finally found the main >> reason: >>>> dependency conflict on Netty. flakka was pulling in 3.8 and zookeeper >> 3.10. >>>> Effect was very subtle, because rarely in some corner cases (but not >>>> always) Netty was deadlocking itself… >>>> >>>> Because of that I would like to enable dependencyConvergence rule in >>>> maven-enforcer-plugin project wide - it catches this error immediately: >>>> >>>> Dependency convergence error for io.netty:netty:3.10.5.Final paths to >>>> dependency are: >>>> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT >>>> +-org.apache.kafka:kafka_2.11:0.9.0.1 >>>> +-org.apache.zookeeper:zookeeper:3.4.10 >>>> +-io.netty:netty:3.10.5.Final >>>> and >>>> +-org.apache.flink:flink-connector-kafka-0.9_2.11:1.4-SNAPSHOT >>>> +-org.apache.flink:flink-runtime_2.11:1.4-SNAPSHOT >>>> +-com.data-artisans:flakka-remote_2.11:2.3-custom >>>> +-io.netty:netty:3.8.0.Final >>>> >>>> Currently this rule fails with multiple errors, but after those lost >>>> couple of days I’m pretty determined to fix all of them “just in case”. >>>> dependencyConvergence rule would protect us in the future against such >>>> nasty subtle bugs. Does anyone have any objections/issues that I’m not >>>> aware of? >>>> >>>> Piotrek >> >> |
Free forum by Nabble | Edit this page |