Hi,
I'm wondering how we want to maintain the ElasticSearch sinks in Flink. Right now, we have two connectors, supporting the 1.x and 2.x versions. There is a pending PR [1] for adding ES 5 support. All three connectors share a lot of code, and there is a pull request [2] improving the ES 2.x connector. If we want to make the ES sinks have proper at least once support, we need to do some more changes, and I would like to do them once for all connectors. I propose to create a new "flink-connector-elasticsearch-base" module with the common code and then create modules for 1.x, 2.x and 5.x. Also, I would merge the documentation pages into one. I would also be open to discuss whether we want to move the EL connectors into Bahir. For now, I'm against that, given that ES is among the more popular connectors in Flink. Regards, Robert [1] https://github.com/apache/flink/pull/2767 [2] https://github.com/apache/flink/pull/2861 |
Thanks for looking into this Robert.
I agree, it would be good to share as much code as possible between the Elasticsearch connectors. +1 for flink-connector-elasticsearch-base +1 for keeping the connectors in Flink 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > Hi, > I'm wondering how we want to maintain the ElasticSearch sinks in Flink. > > Right now, we have two connectors, supporting the 1.x and 2.x versions. > There is a pending PR [1] for adding ES 5 support. > > All three connectors share a lot of code, and there is a pull request [2] > improving the ES 2.x connector. If we want to make the ES sinks have proper > at least once support, we need to do some more changes, and I would like to > do them once for all connectors. > I propose to create a new "flink-connector-elasticsearch-base" module with > the common code and then create modules for 1.x, 2.x and 5.x. > Also, I would merge the documentation pages into one. > > I would also be open to discuss whether we want to move the EL connectors > into Bahir. > For now, I'm against that, given that ES is among the more popular > connectors in Flink. > > > Regards, > Robert > > [1] https://github.com/apache/flink/pull/2767 > [2] https://github.com/apache/flink/pull/2861 > |
Thanks for starting the discussion Robert.
I agree that we should try to treat the Elasticsearch connectors in a similar fashion to the multiple Kafka connector versions. +1 for flink-connector-elasticsearch-base +1 to consolidating the documents +1 for keeping the ES connector in Flink I’d also like to proceed with this with https://github.com/apache/flink/pull/2767, by opening another new PR based on #2767 that also refactors the ES connectors & consolidates documentation. After that, we can proceed with https://github.com/apache/flink/pull/2861 for the proper at least once support across all ES versions. - Gordon On December 19, 2016 at 11:58:03 PM, Fabian Hueske ([hidden email]) wrote: Thanks for looking into this Robert. I agree, it would be good to share as much code as possible between the Elasticsearch connectors. +1 for flink-connector-elasticsearch-base +1 for keeping the connectors in Flink 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > Hi, > I'm wondering how we want to maintain the ElasticSearch sinks in Flink. > > Right now, we have two connectors, supporting the 1.x and 2.x versions. > There is a pending PR [1] for adding ES 5 support. > > All three connectors share a lot of code, and there is a pull request [2] > improving the ES 2.x connector. If we want to make the ES sinks have proper > at least once support, we need to do some more changes, and I would like to > do them once for all connectors. > I propose to create a new "flink-connector-elasticsearch-base" module with > the common code and then create modules for 1.x, 2.x and 5.x. > Also, I would merge the documentation pages into one. > > I would also be open to discuss whether we want to move the EL connectors > into Bahir. > For now, I'm against that, given that ES is among the more popular > connectors in Flink. > > > Regards, > Robert > > [1] https://github.com/apache/flink/pull/2767 > [2] https://github.com/apache/flink/pull/2861 > |
I think also https://issues.apache.org/jira/browse/FLINK-5353 should be
solved sooner or later... Best, Flavio On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <[hidden email]> wrote: > Thanks for starting the discussion Robert. > > I agree that we should try to treat the Elasticsearch connectors in a > similar fashion to the multiple Kafka connector versions. > > +1 for flink-connector-elasticsearch-base > +1 to consolidating the documents > +1 for keeping the ES connector in Flink > > I’d also like to proceed with this with https://github.com/apache/ > flink/pull/2767, by opening another new PR based on #2767 that also > refactors the ES connectors & consolidates documentation. > After that, we can proceed with https://github.com/apache/flink/pull/2861 for > the proper at least once support across all ES versions. > > - Gordon > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske ([hidden email]) > wrote: > > Thanks for looking into this Robert. > I agree, it would be good to share as much code as possible between the > Elasticsearch connectors. > > +1 for flink-connector-elasticsearch-base > +1 for keeping the connectors in Flink > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > > > Hi, > > I'm wondering how we want to maintain the ElasticSearch sinks in Flink. > > > > Right now, we have two connectors, supporting the 1.x and 2.x versions. > > There is a pending PR [1] for adding ES 5 support. > > > > All three connectors share a lot of code, and there is a pull request [2] > > improving the ES 2.x connector. If we want to make the ES sinks have > proper > > at least once support, we need to do some more changes, and I would like > to > > do them once for all connectors. > > I propose to create a new "flink-connector-elasticsearch-base" module > with > > the common code and then create modules for 1.x, 2.x and 5.x. > > Also, I would merge the documentation pages into one. > > > > I would also be open to discuss whether we want to move the EL connectors > > into Bahir. > > For now, I'm against that, given that ES is among the more popular > > connectors in Flink. > > > > > > Regards, > > Robert > > > > [1] https://github.com/apache/flink/pull/2767 > > [2] https://github.com/apache/flink/pull/2861 > > > |
Thanks for bringing this up again Flavio.
The good thing about a common base for all ES connectors is that we can fix the issue for all ES versions (ideally). Thanks a lot for taking care of my proposed changes Gordon! On Mon, Jan 9, 2017 at 12:47 PM, Flavio Pompermaier <[hidden email]> wrote: > I think also https://issues.apache.org/jira/browse/FLINK-5353 should be > solved sooner or later... > > Best, > Flavio > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <[hidden email]> wrote: > > > Thanks for starting the discussion Robert. > > > > I agree that we should try to treat the Elasticsearch connectors in a > > similar fashion to the multiple Kafka connector versions. > > > > +1 for flink-connector-elasticsearch-base > > +1 to consolidating the documents > > +1 for keeping the ES connector in Flink > > > > I’d also like to proceed with this with https://github.com/apache/ > > flink/pull/2767, by opening another new PR based on #2767 that also > > refactors the ES connectors & consolidates documentation. > > After that, we can proceed with https://github.com/apache/flin > k/pull/2861 for > > the proper at least once support across all ES versions. > > > > - Gordon > > > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske ([hidden email]) > > wrote: > > > > Thanks for looking into this Robert. > > I agree, it would be good to share as much code as possible between the > > Elasticsearch connectors. > > > > +1 for flink-connector-elasticsearch-base > > +1 for keeping the connectors in Flink > > > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > > > > > Hi, > > > I'm wondering how we want to maintain the ElasticSearch sinks in Flink. > > > > > > Right now, we have two connectors, supporting the 1.x and 2.x versions. > > > There is a pending PR [1] for adding ES 5 support. > > > > > > All three connectors share a lot of code, and there is a pull request > [2] > > > improving the ES 2.x connector. If we want to make the ES sinks have > > proper > > > at least once support, we need to do some more changes, and I would > like > > to > > > do them once for all connectors. > > > I propose to create a new "flink-connector-elasticsearch-base" module > > with > > > the common code and then create modules for 1.x, 2.x and 5.x. > > > Also, I would merge the documentation pages into one. > > > > > > I would also be open to discuss whether we want to move the EL > connectors > > > into Bahir. > > > For now, I'm against that, given that ES is among the more popular > > > connectors in Flink. > > > > > > > > > Regards, > > > Robert > > > > > > [1] https://github.com/apache/flink/pull/2767 > > > [2] https://github.com/apache/flink/pull/2861 > > > > > > |
+1 for keeping the ES connector in Flink
On Mon, Jan 9, 2017 at 2:53 PM, Robert Metzger <[hidden email]> wrote: > Thanks for bringing this up again Flavio. > > The good thing about a common base for all ES connectors is that we can fix > the issue for all ES versions (ideally). > > Thanks a lot for taking care of my proposed changes Gordon! > > > On Mon, Jan 9, 2017 at 12:47 PM, Flavio Pompermaier <[hidden email]> > wrote: > > > I think also https://issues.apache.org/jira/browse/FLINK-5353 should be > > solved sooner or later... > > > > Best, > > Flavio > > > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <[hidden email]> wrote: > > > > > Thanks for starting the discussion Robert. > > > > > > I agree that we should try to treat the Elasticsearch connectors in a > > > similar fashion to the multiple Kafka connector versions. > > > > > > +1 for flink-connector-elasticsearch-base > > > +1 to consolidating the documents > > > +1 for keeping the ES connector in Flink > > > > > > I’d also like to proceed with this with https://github.com/apache/ > > > flink/pull/2767, by opening another new PR based on #2767 that also > > > refactors the ES connectors & consolidates documentation. > > > After that, we can proceed with https://github.com/apache/flin > > k/pull/2861 for > > > the proper at least once support across all ES versions. > > > > > > - Gordon > > > > > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske ([hidden email]) > > > wrote: > > > > > > Thanks for looking into this Robert. > > > I agree, it would be good to share as much code as possible between the > > > Elasticsearch connectors. > > > > > > +1 for flink-connector-elasticsearch-base > > > +1 for keeping the connectors in Flink > > > > > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > > > > > > > Hi, > > > > I'm wondering how we want to maintain the ElasticSearch sinks in > Flink. > > > > > > > > Right now, we have two connectors, supporting the 1.x and 2.x > versions. > > > > There is a pending PR [1] for adding ES 5 support. > > > > > > > > All three connectors share a lot of code, and there is a pull request > > [2] > > > > improving the ES 2.x connector. If we want to make the ES sinks have > > > proper > > > > at least once support, we need to do some more changes, and I would > > like > > > to > > > > do them once for all connectors. > > > > I propose to create a new "flink-connector-elasticsearch-base" > module > > > with > > > > the common code and then create modules for 1.x, 2.x and 5.x. > > > > Also, I would merge the documentation pages into one. > > > > > > > > I would also be open to discuss whether we want to move the EL > > connectors > > > > into Bahir. > > > > For now, I'm against that, given that ES is among the more popular > > > > connectors in Flink. > > > > > > > > > > > > Regards, > > > > Robert > > > > > > > > [1] https://github.com/apache/flink/pull/2767 > > > > [2] https://github.com/apache/flink/pull/2861 > > > > > > > > > > |
Hi,
As part of restructuring the ES connectors, I think it also a very good timing to unify how our ES connectors are used. For the 1.x connector, users provide a “IndexRequestBuilder” that was only capable of indexing a single document for each record. For the 2.x and pending 5.x connector, the API was improved to let users provide a “ElasticsearchSinkFunction” that can yield many ES actions (modify, index, delete, etc.) for each record. I’m not sure of the history on this, but it seems like the API improvement to 2.x+ wasn’t reflected also on the 1.x connector at the time. I’d like to deprecate the “IndexRequestBuilder” constructor variant in 1.x, and let it have a new constructor that takes “ElasticsearchSinkFunction” like the other versions. This will also have the advantage of much cleaner structure across the ES connector modules. What do you think? - Gordon On January 9, 2017 at 10:50:55 PM, Kostas Tzoumas ([hidden email]) wrote: +1 for keeping the ES connector in Flink On Mon, Jan 9, 2017 at 2:53 PM, Robert Metzger <[hidden email]> wrote: > Thanks for bringing this up again Flavio. > > The good thing about a common base for all ES connectors is that we can fix > the issue for all ES versions (ideally). > > Thanks a lot for taking care of my proposed changes Gordon! > > > On Mon, Jan 9, 2017 at 12:47 PM, Flavio Pompermaier <[hidden email]> > wrote: > > > I think also https://issues.apache.org/jira/browse/FLINK-5353 should be > > solved sooner or later... > > > > Best, > > Flavio > > > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <[hidden email]> wrote: > > > > > Thanks for starting the discussion Robert. > > > > > > I agree that we should try to treat the Elasticsearch connectors in a > > > similar fashion to the multiple Kafka connector versions. > > > > > > +1 for flink-connector-elasticsearch-base > > > +1 to consolidating the documents > > > +1 for keeping the ES connector in Flink > > > > > > I’d also like to proceed with this with https://github.com/apache/ > > > flink/pull/2767, by opening another new PR based on #2767 that also > > > refactors the ES connectors & consolidates documentation. > > > After that, we can proceed with https://github.com/apache/flin > > k/pull/2861 for > > > the proper at least once support across all ES versions. > > > > > > - Gordon > > > > > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske ([hidden email]) > > > wrote: > > > > > > Thanks for looking into this Robert. > > > I agree, it would be good to share as much code as possible between the > > > Elasticsearch connectors. > > > > > > +1 for flink-connector-elasticsearch-base > > > +1 for keeping the connectors in Flink > > > > > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > > > > > > > Hi, > > > > I'm wondering how we want to maintain the ElasticSearch sinks in > Flink. > > > > > > > > Right now, we have two connectors, supporting the 1.x and 2.x > versions. > > > > There is a pending PR [1] for adding ES 5 support. > > > > > > > > All three connectors share a lot of code, and there is a pull request > > [2] > > > > improving the ES 2.x connector. If we want to make the ES sinks have > > > proper > > > > at least once support, we need to do some more changes, and I would > > like > > > to > > > > do them once for all connectors. > > > > I propose to create a new "flink-connector-elasticsearch-base" > module > > > with > > > > the common code and then create modules for 1.x, 2.x and 5.x. > > > > Also, I would merge the documentation pages into one. > > > > > > > > I would also be open to discuss whether we want to move the EL > > connectors > > > > into Bahir. > > > > For now, I'm against that, given that ES is among the more popular > > > > connectors in Flink. > > > > > > > > > > > > Regards, > > > > Robert > > > > > > > > [1] https://github.com/apache/flink/pull/2767 > > > > [2] https://github.com/apache/flink/pull/2861 > > > > > > > > > > |
I definitively agree, if we can cover the functionality with all ES
dependency versions. On Tue, Jan 10, 2017 at 6:50 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote: > Hi, > > As part of restructuring the ES connectors, I think it also a very good > timing to unify how our ES connectors are used. > > For the 1.x connector, users provide a “IndexRequestBuilder” that was only > capable of indexing a single document for each record. > For the 2.x and pending 5.x connector, the API was improved to let users > provide a “ElasticsearchSinkFunction” that can yield many ES actions > (modify, index, delete, etc.) for each record. > > I’m not sure of the history on this, but it seems like the API improvement > to 2.x+ wasn’t reflected also on the 1.x connector at the time. > > I’d like to deprecate the “IndexRequestBuilder” constructor variant in > 1.x, and let it have a new constructor that takes > “ElasticsearchSinkFunction” like the other versions. > This will also have the advantage of much cleaner structure across the ES > connector modules. What do you think? > > - Gordon > > On January 9, 2017 at 10:50:55 PM, Kostas Tzoumas ([hidden email]) > wrote: > > +1 for keeping the ES connector in Flink > > On Mon, Jan 9, 2017 at 2:53 PM, Robert Metzger <[hidden email]> > wrote: > > > Thanks for bringing this up again Flavio. > > > > The good thing about a common base for all ES connectors is that we can > fix > > the issue for all ES versions (ideally). > > > > Thanks a lot for taking care of my proposed changes Gordon! > > > > > > On Mon, Jan 9, 2017 at 12:47 PM, Flavio Pompermaier < > [hidden email]> > > wrote: > > > > > I think also https://issues.apache.org/jira/browse/FLINK-5353 should > be > > > solved sooner or later... > > > > > > Best, > > > Flavio > > > > > > On 9 Jan 2017 12:24, "Tzu-Li (Gordon) Tai" <[hidden email]> > wrote: > > > > > > > Thanks for starting the discussion Robert. > > > > > > > > I agree that we should try to treat the Elasticsearch connectors in a > > > > similar fashion to the multiple Kafka connector versions. > > > > > > > > +1 for flink-connector-elasticsearch-base > > > > +1 to consolidating the documents > > > > +1 for keeping the ES connector in Flink > > > > > > > > I’d also like to proceed with this with https://github.com/apache/ > > > > flink/pull/2767, by opening another new PR based on #2767 that also > > > > refactors the ES connectors & consolidates documentation. > > > > After that, we can proceed with https://github.com/apache/flin > > > k/pull/2861 for > > > > the proper at least once support across all ES versions. > > > > > > > > - Gordon > > > > > > > > On December 19, 2016 at 11:58:03 PM, Fabian Hueske ( > [hidden email]) > > > > wrote: > > > > > > > > Thanks for looking into this Robert. > > > > I agree, it would be good to share as much code as possible between > the > > > > Elasticsearch connectors. > > > > > > > > +1 for flink-connector-elasticsearch-base > > > > +1 for keeping the connectors in Flink > > > > > > > > 2016-12-19 16:46 GMT+01:00 Robert Metzger <[hidden email]>: > > > > > > > > > Hi, > > > > > I'm wondering how we want to maintain the ElasticSearch sinks in > > Flink. > > > > > > > > > > Right now, we have two connectors, supporting the 1.x and 2.x > > versions. > > > > > There is a pending PR [1] for adding ES 5 support. > > > > > > > > > > All three connectors share a lot of code, and there is a pull > request > > > [2] > > > > > improving the ES 2.x connector. If we want to make the ES sinks > have > > > > proper > > > > > at least once support, we need to do some more changes, and I would > > > like > > > > to > > > > > do them once for all connectors. > > > > > I propose to create a new "flink-connector-elasticsearch-base" > > module > > > > with > > > > > the common code and then create modules for 1.x, 2.x and 5.x. > > > > > Also, I would merge the documentation pages into one. > > > > > > > > > > I would also be open to discuss whether we want to move the EL > > > connectors > > > > > into Bahir. > > > > > For now, I'm against that, given that ES is among the more popular > > > > > connectors in Flink. > > > > > > > > > > > > > > > Regards, > > > > > Robert > > > > > > > > > > [1] https://github.com/apache/flink/pull/2767 > > > > > [2] https://github.com/apache/flink/pull/2861 > > > > > > > > > > > > > > > |
Free forum by Nabble | Edit this page |