Move Stream Connector Jars from "lib" to Client JARs

[Stephan Ewen]

Hi everyone!

How do you feel about changing the way we handle the streaming connectors?

Right now, the connectors and all dependencies are put into the lib folder
and are part of the system at startup time. This is a large bunch of
dependencies, and they may actually conflict with the dependencies of
custom connectors (or example with a different version of RabbitMQ or so).

We could fix that, if we remove the dependencies from the lib folder and
set up archetypes that build fat jars with the dependencies. That way, each
job (with its custom class loader) will gets the dependencies it needs and
will not see all the other (potentially conflicting ones) in the namespace.

Greetings,
Stephan

[Márton Balassi]

+1 That is a nice solution.

Assigned it to myself. [1]

[1] https://issues.apache.org/jira/browse/FLINK-1348

On Mon, Jan 5, 2015 at 10:34 AM, Stephan Ewen <[hidden email]> wrote:

> Hi everyone!
>
> How do you feel about changing the way we handle the streaming connectors?
>
> Right now, the connectors and all dependencies are put into the lib folder
> and are part of the system at startup time. This is a large bunch of
> dependencies, and they may actually conflict with the dependencies of
> custom connectors (or example with a different version of RabbitMQ or so).
>
> We could fix that, if we remove the dependencies from the lib folder and
> set up archetypes that build fat jars with the dependencies. That way, each
> job (with its custom class loader) will gets the dependencies it needs and
> will not see all the other (potentially conflicting ones) in the namespace.
>
> Greetings,
> Stephan
>