I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1]. But I recall recently seeing a message on the dev list about moving some of the third party connectors out of Flink as devs were having problems maintaining them. So, is this the sort of thing I should contribute to Flink or is the plan to house these types of things elsewhere?
Alan. 1. https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest |
Hi Alan,
Your contribution is more than welcome. It would be a great addition to flink-streaming-connectors. At some point we might move some of these to a "Flink Packages" repository, similarly to the Spark approach, but currently the best place to have them is the internal connectors. Robert in the previous mail was referring to some file system based streaming sources, those are tricky to maintain as they have many overloaded versions. Those will be moved to a different place within the Flink repo. Best, Marton On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote: > I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1]. > But I recall recently seeing a message on the dev list about moving some > of the third party connectors out of Flink as devs were having problems > maintaining them. So, is this the sort of thing I should contribute to > Flink or is the plan to house these types of things elsewhere? > > Alan. > > 1. https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest |
Hi Alan,
since you are a committer of Flink I don't see any issues merging such a connector to Flink. The reason why we didn't move the connectors out of the Flink repository was the additional overhead for the CI system, Maven files, Apache release process etc. so we decided to maintain connectors within Flink for now. Please let us know if you have any questions while implementing the sink. On Thu, Jul 7, 2016 at 11:17 AM, Márton Balassi <[hidden email]> wrote: > Hi Alan, > > Your contribution is more than welcome. It would be a great addition to > flink-streaming-connectors. At some point we might move some of these to a > "Flink Packages" repository, similarly to the Spark approach, but currently > the best place to have them is the internal connectors. > > Robert in the previous mail was referring to some file system based > streaming sources, those are tricky to maintain as they have many > overloaded versions. Those will be moved to a different place within the > Flink repo. > > Best, > > Marton > > On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote: > > > I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1]. > > But I recall recently seeing a message on the dev list about moving some > > of the third party connectors out of Flink as devs were having problems > > maintaining them. So, is this the sort of thing I should contribute to > > Flink or is the plan to house these types of things elsewhere? > > > > Alan. > > > > 1. > https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest > |
In reply to this post by Alan Gates-2
Hey Alan,
as Marton said your contribution is more than welcome. :-) The discussion around moving some contributions outside of the main repository did never come to a final conclusion. Therefore, we currently have most of the connectors inside of main Flink repo. As long as there is no concrete proposal or guide line about what we keep in Flink and what not, I think it's best to contribute it directly to Flink. – Ufuk On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote: > I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1]. But I recall recently seeing a message on the dev list about moving some of the third party connectors out of Flink as devs were having problems maintaining them. So, is this the sort of thing I should contribute to Flink or is the plan to house these types of things elsewhere? > > Alan. > > 1. https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest |
Moreover, that would reduce the build duration.
Le 7 juil. 2016 11:25, "Ufuk Celebi" <[hidden email]> a écrit : > Hey Alan, > > as Marton said your contribution is more than welcome. :-) > > The discussion around moving some contributions outside of the main > repository did never come to a final conclusion. Therefore, we > currently have most of the connectors inside of main Flink repo. As > long as there is no concrete proposal or guide line about what we keep > in Flink and what not, I think it's best to contribute it directly to > Flink. > > – Ufuk > > > On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote: > > I’d like to work on creating a Flink Sink for Hive’s streaming > ingest[1]. But I recall recently seeing a message on the dev list about > moving some of the third party connectors out of Flink as devs were having > problems maintaining them. So, is this the sort of thing I should > contribute to Flink or is the plan to house these types of things elsewhere? > > > > Alan. > > > > 1. > https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest > |
Free forum by Nabble | Edit this page |