Re: [DISCUSS] Drop connectors for 5.x and restart the flink es source connector

[Jark Wu-2]

I'm fine with dropping support for es5.

forward to dev@.

Best,
Jark



On Fri, 19 Jun 2020 at 15:46, jackylau <[hidden email]> wrote:

> Hi all:
>      when i coding the es source connector here
>
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> for the elasticsearch source connector. The doc is here
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> ,and
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and>
> i find  problem of SearchHits at
>
> *currentScrollWindowHits = searchResponse.getHits().getHits()
> *
> And the SearchHits is a Interface in es5 but it is a class in es 6,7 and if
> i use it in flink es connector base(the es dependency is 5). it will throw
> this.
> *Caused by: java.lang.IncompatibleClassChangeError: Found class
> org.elasticsearch.search.SearchHits, but interface was expected.***
>
> To fix it : we can do this ways
>
> 1) move the logic to ApiCallBridge such as define ElasticsearchResponse or
> Tuple2<String, String[]>, but it wll make the code weirdly
>
> class ElasticsearchResponse
> {
>     String scroll;
>     String[] result // convert ervery es connector
> searchResponse.getHits().getHits() to this result
> }
>
>
> if user want to add some thing, it will need modify this
>
>
> 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> version to 6 and drop flink-es-connector-5. And i found this discussion of
> dropping es connector 2 and 5 here
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> .
>
>  The es5 connector just support DataStream api currently .And Is it
> possible
> to drop es5 connector and upgrade es-connector-base to es6?
>
> I am looking forward all your response
> Best !
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

[Robert Metzger]

The discussion on dropping the ES5 connector was not conclusive, when we
discussed it in February 2020. We wanted to revisit it for the 1.12 release.

From maven central, we have the following download numbers
ES2: 500 downloads
ES5: 10500 downloads (the es5_2.10:1.3.1 had 8000 downloads last month. I
there's a CI system or something downloading all these)
ES6: 4200 downloads
ES7: 1800 downloads

For 1.10.0 we had the following numbers:
ES5: 500
ES6: 525
ES7: 840

Based on these numbers, I would advise against removing the ES5 connector
for the 1.12 release.



On Fri, Jun 19, 2020 at 9:53 AM Jark Wu <[hidden email]> wrote:

> I'm fine with dropping support for es5.
>
> forward to dev@.
>
> Best,
> Jark
>
>
>
> On Fri, 19 Jun 2020 at 15:46, jackylau <[hidden email]> wrote:
>
> > Hi all:
> >      when i coding the es source connector here
> >
> >
> https://github.com/liuyongvs/flink/commit/c397a759d05956629a27bf850458dd4e70330189
> > for the elasticsearch source connector. The doc is here
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector
> > ,and
> > <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-127%3A+Support+Elasticsearch+Source+Connector,and
> >
> > i find  problem of SearchHits at
> >
> > *currentScrollWindowHits = searchResponse.getHits().getHits()
> > *
> > And the SearchHits is a Interface in es5 but it is a class in es 6,7 and
> if
> > i use it in flink es connector base(the es dependency is 5). it will
> throw
> > this.
> > *Caused by: java.lang.IncompatibleClassChangeError: Found class
> > org.elasticsearch.search.SearchHits, but interface was expected.***
> >
> > To fix it : we can do this ways
> >
> > 1) move the logic to ApiCallBridge such as define ElasticsearchResponse
> or
> > Tuple2<String, String[]>, but it wll make the code weirdly
> >
> > class ElasticsearchResponse
> > {
> >     String scroll;
> >     String[] result // convert ervery es connector
> > searchResponse.getHits().getHits() to this result
> > }
> >
> >
> > if user want to add some thing, it will need modify this
> >
> >
> > 2) just support es 6,7 and upgrade flink-es-connector-base es dependency
> > version to 6 and drop flink-es-connector-5. And i found this discussion
> of
> > dropping es connector 2 and 5 here
> >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-connectors-for-Elasticsearch-2-x-and-5-x-td32662.html
> > .
> >
> >  The es5 connector just support DataStream api currently .And Is it
> > possible
> > to drop es5 connector and upgrade es-connector-base to es6?
> >
> > I am looking forward all your response
> > Best !
> >
> >
> >
> > --
> > Sent from:
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
> >
>