[DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

[dwysakowicz]

Hi all,

As described in this https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced in 5.2.

Both versions are already long eol: https://www.elastic.co/support/eol

I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.

What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?

Best,

Dawid

[Robert Metzger]

Thanks for starting this discussion!

+1 to drop both

On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]>
wrote:

> Hi all,
>
> As described in this https://issues.apache.org/jira/browse/FLINK-11720
> ticket our elasticsearch 5.x connector does not work out of the box on
> some systems and requires a version bump. This also happens for our e2e.
> We cannot bump the version in es 5.x connector, because 5.x connector
> shares a common class with 2.x that uses an API that was replaced in 5.2.
>
> Both versions are already long eol: https://www.elastic.co/support/eol
>
> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> both of them, I would strongly suggest dropping at least 2.x connector
> and update the 5.x line to a working es client module.
>
> What do you think? Should we drop both versions? Drop only the 2.x
> connector? Or keep them both?
>
> Best,
>
> Dawid
>
>
>

[Itamar Syn-Hershko]

+1 from dropping old versions because of jar hells etc. However, in the
wild there are still a lot of 2.x clusters and definitely 5.x clusters that
are having a hard time upgrading. We know because we assist those on a
daily basis.

It is very easy to create an HTTP based connector that works with all ES
versions, though. As Elasticsearch consultants and experts we have done
that many times before. For example see this simplified client that has
zero dependencies and can be easily brought in to Flink to use as a sink
for all ES versions:
https://github.com/BigDataBoutique/log4j2-elasticsearch-http/blob/master/src/main/java/com/bigdataboutique/logging/log4j2/ElasticsearchHttpClient.java

Will be happy to assist in such effort

On Mon, Feb 10, 2020 at 3:45 PM Dawid Wysakowicz <[hidden email]>
wrote:

> Hi all,
>
> As described in this https://issues.apache.org/jira/browse/FLINK-11720
> ticket our elasticsearch 5.x connector does not work out of the box on
> some systems and requires a version bump. This also happens for our e2e.
> We cannot bump the version in es 5.x connector, because 5.x connector
> shares a common class with 2.x that uses an API that was replaced in 5.2.
>
> Both versions are already long eol: https://www.elastic.co/support/eol
>
> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> both of them, I would strongly suggest dropping at least 2.x connector
> and update the 5.x line to a working es client module.
>
> What do you think? Should we drop both versions? Drop only the 2.x
> connector? Or keep them both?
>
> Best,
>
> Dawid
>
>
>

--

[image: logo] <https://bigdataboutique.com/>
Itamar Syn-Hershko
CTO, Founder

[hidden email]
https://bigdataboutique.com
<https://www.linkedin.com/in/itamar-syn-hershko-78b25013>
<https://twitter.com/synhershko>
<https://www.youtube.com/channel/UCBHr7lM2u6SCWPJvcKug-Yg>

[Flavio Pompermaier]

+1 for dropping all Elasticsearch connectors < 6.x

On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]>
wrote:

> Hi all,
>
> As described in this https://issues.apache.org/jira/browse/FLINK-11720
> ticket our elasticsearch 5.x connector does not work out of the box on
> some systems and requires a version bump. This also happens for our e2e.
> We cannot bump the version in es 5.x connector, because 5.x connector
> shares a common class with 2.x that uses an API that was replaced in 5.2.
>
> Both versions are already long eol: https://www.elastic.co/support/eol
>
> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> both of them, I would strongly suggest dropping at least 2.x connector
> and update the 5.x line to a working es client module.
>
> What do you think? Should we drop both versions? Drop only the 2.x
> connector? Or keep them both?
>
> Best,
>
> Dawid
>
>

[Benchao Li]

+1 for dropping 2.x - 5.x.

FYI currently only 6.x and 7.x ES Connectors are supported by table api.

Flavio Pompermaier <[hidden email]> 于2020年2月10日周一 下午10:03写道：

> +1 for dropping all Elasticsearch connectors < 6.x
>
> On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]>
> wrote:
>
> > Hi all,
> >
> > As described in this https://issues.apache.org/jira/browse/FLINK-11720
> > ticket our elasticsearch 5.x connector does not work out of the box on
> > some systems and requires a version bump. This also happens for our e2e.
> > We cannot bump the version in es 5.x connector, because 5.x connector
> > shares a common class with 2.x that uses an API that was replaced in 5.2.
> >
> > Both versions are already long eol: https://www.elastic.co/support/eol
> >
> > I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> > both of them, I would strongly suggest dropping at least 2.x connector
> > and update the 5.x line to a working es client module.
> >
> > What do you think? Should we drop both versions? Drop only the 2.x
> > connector? Or keep them both?
> >
> > Best,
> >
> > Dawid
> >
> >
>

--

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:[phone]
Email: [hidden email]; [hidden email]

[Aljoscha Krettek-2]

+1 for dropping them, this stuff is quite old by now.

On 10.02.20 15:04, Benchao Li wrote:

> +1 for dropping 2.x - 5.x.
>
> FYI currently only 6.x and 7.x ES Connectors are supported by table api.
>
> Flavio Pompermaier <[hidden email]> 于2020年2月10日周一 下午10:03写道：
>
>> +1 for dropping all Elasticsearch connectors < 6.x
>>
>> On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz <[hidden email]>
>> wrote:
>>
>>> Hi all,
>>>
>>> As described in this https://issues.apache.org/jira/browse/FLINK-11720
>>> ticket our elasticsearch 5.x connector does not work out of the box on
>>> some systems and requires a version bump. This also happens for our e2e.
>>> We cannot bump the version in es 5.x connector, because 5.x connector
>>> shares a common class with 2.x that uses an API that was replaced in 5.2.
>>>
>>> Both versions are already long eol: https://www.elastic.co/support/eol
>>>
>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
>>> both of them, I would strongly suggest dropping at least 2.x connector
>>> and update the 5.x line to a working es client module.
>>>
>>> What do you think? Should we drop both versions? Drop only the 2.x
>>> connector? Or keep them both?
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>>
>>
>
>

[Danny Chan]

5.x seems to have a lot of users, is the 6.x completely compatible with 5.x ~

Best,
Danny Chan
在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz <[hidden email]>，写道：

> Hi all,
>
> As described in this https://issues.apache.org/jira/browse/FLINK-11720
> ticket our elasticsearch 5.x connector does not work out of the box on
> some systems and requires a version bump. This also happens for our e2e.
> We cannot bump the version in es 5.x connector, because 5.x connector
> shares a common class with 2.x that uses an API that was replaced in 5.2.
>
> Both versions are already long eol: https://www.elastic.co/support/eol
>
> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> both of them, I would strongly suggest dropping at least 2.x connector
> and update the 5.x line to a working es client module.
>
> What do you think? Should we drop both versions? Drop only the 2.x
> connector? Or keep them both?
>
> Best,
>
> Dawid
>
>

[Stephan Ewen]

+1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
for that one).

@Itamar - if you would be interested in contributing a "universal" or
"cross version" ES connector, that could be very interesting. Do you know
if there are known performance issues or feature restrictions with that
approach?
@dawid what do you think about that?


On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]> wrote:

> 5.x seems to have a lot of users, is the 6.x completely compatible with
> 5.x ~
>
> Best,
> Danny Chan
> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz <[hidden email]>，写道：
> > Hi all,
> >
> > As described in this https://issues.apache.org/jira/browse/FLINK-11720
> > ticket our elasticsearch 5.x connector does not work out of the box on
> > some systems and requires a version bump. This also happens for our e2e.
> > We cannot bump the version in es 5.x connector, because 5.x connector
> > shares a common class with 2.x that uses an API that was replaced in 5.2.
> >
> > Both versions are already long eol: https://www.elastic.co/support/eol
> >
> > I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
> > both of them, I would strongly suggest dropping at least 2.x connector
> > and update the 5.x line to a working es client module.
> >
> > What do you think? Should we drop both versions? Drop only the 2.x
> > connector? Or keep them both?
> >
> > Best,
> >
> > Dawid
> >
> >
>

[Chesnay Schepler-3]

I suppose the downside in an HTTP ES sink is that you don't get _any_
form of high-level API from ES, and we'd have to manually build an HTTP
request that matches the ES format. Of course you also lose any
client-side verification that the clients did, if there is any (but I
guess the API itself prevented certain errors).

On 11/02/2020 09:32, Stephan Ewen wrote:

> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
> for that one).
>
> @Itamar - if you would be interested in contributing a "universal" or
> "cross version" ES connector, that could be very interesting. Do you know
> if there are known performance issues or feature restrictions with that
> approach?
> @dawid what do you think about that?
>
>
> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]> wrote:
>
>> 5.x seems to have a lot of users, is the 6.x completely compatible with
>> 5.x ~
>>
>> Best,
>> Danny Chan
>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz <[hidden email]>，写道：
>>> Hi all,
>>>
>>> As described in this https://issues.apache.org/jira/browse/FLINK-11720
>>> ticket our elasticsearch 5.x connector does not work out of the box on
>>> some systems and requires a version bump. This also happens for our e2e.
>>> We cannot bump the version in es 5.x connector, because 5.x connector
>>> shares a common class with 2.x that uses an API that was replaced in 5.2.
>>>
>>> Both versions are already long eol: https://www.elastic.co/support/eol
>>>
>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to drop
>>> both of them, I would strongly suggest dropping at least 2.x connector
>>> and update the 5.x line to a working es client module.
>>>
>>> What do you think? Should we drop both versions? Drop only the 2.x
>>> connector? Or keep them both?
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>>

[dwysakowicz]

Sorry for late reply,

@all I think there is a general consensus that we want to drop ES 2.x
support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
track it.


@Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
Client. ES promises to maintain compatibility of it with any newer minor
version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3 etc.

ES provides also a low level rest client which does not include any
direct es dependencies and can work with any version of ES. It does not
provide any marshalling unmarshalling or higher level features as
Chesnay said.

Correct me if I am wrong @Itamar but your HTTP client is a simplified
version of the ES's high level rest client with a subset of its
features. I think it will still have the same problems as ES's High
Level Rest Client's because ES does not guarantee that newer message
formats will be compatible with older versions of ES or that message
formats are compatible across major versions at all.


@Stephan @Danny As for the 5.x connector. Any ideas how can we get
user's feedback about it? I cross posted on the user mailing list with
no luck so far. Personally I would be in favor of dropping the
connector. Worst case scenario users still have the possibility of
building the connector themselves from source with just bumping the
flink's versions. As far as I can tell there were no changes to the code
base for quite some time.

Best,

Dawid

On 11/02/2020 10:46, Chesnay Schepler wrote:

> I suppose the downside in an HTTP ES sink is that you don't get _any_
> form of high-level API from ES, and we'd have to manually build an
> HTTP request that matches the ES format. Of course you also lose any
> client-side verification that the clients did, if there is any (but I
> guess the API itself prevented certain errors).
>
> On 11/02/2020 09:32, Stephan Ewen wrote:
>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
>> for that one).
>>
>> @Itamar - if you would be interested in contributing a "universal" or
>> "cross version" ES connector, that could be very interesting. Do you
>> know
>> if there are known performance issues or feature restrictions with that
>> approach?
>> @dawid what do you think about that?
>>
>>
>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]> wrote:
>>
>>> 5.x seems to have a lot of users, is the 6.x completely compatible with
>>> 5.x ~
>>>
>>> Best,
>>> Danny Chan
>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
>>> <[hidden email]>，写道：
>>>> Hi all,
>>>>
>>>> As described in this https://issues.apache.org/jira/browse/FLINK-11720
>>>> ticket our elasticsearch 5.x connector does not work out of the box on
>>>> some systems and requires a version bump. This also happens for our
>>>> e2e.
>>>> We cannot bump the version in es 5.x connector, because 5.x connector
>>>> shares a common class with 2.x that uses an API that was replaced
>>>> in 5.2.
>>>>
>>>> Both versions are already long eol: https://www.elastic.co/support/eol
>>>>
>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to
>>>> drop
>>>> both of them, I would strongly suggest dropping at least 2.x connector
>>>> and update the 5.x line to a working es client module.
>>>>
>>>> What do you think? Should we drop both versions? Drop only the 2.x
>>>> connector? Or keep them both?
>>>>
>>>> Best,
>>>>
>>>> Dawid
>>>>
>>>>
>

[Robert Metzger]

The ES5 connector is causing some problems on the CI system. It would be
nice if we could make a decision here soon. I don't want to invest time
into fixing it, if we are going to remove it.

I'm still in favor of removing it. If we see that there's demand for the
5.x connector after the 1.11 release, somebody can take the source and
contribute it to Apache Bahir or a GitHub account and then posts it to
flink-packages.org.


On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz <[hidden email]>
wrote:

> Sorry for late reply,
>
> @all I think there is a general consensus that we want to drop ES 2.x
> support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
> track it.
>
>
> @Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
> Client. ES promises to maintain compatibility of it with any newer minor
> version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
> etc.
>
> ES provides also a low level rest client which does not include any
> direct es dependencies and can work with any version of ES. It does not
> provide any marshalling unmarshalling or higher level features as
> Chesnay said.
>
> Correct me if I am wrong @Itamar but your HTTP client is a simplified
> version of the ES's high level rest client with a subset of its
> features. I think it will still have the same problems as ES's High
> Level Rest Client's because ES does not guarantee that newer message
> formats will be compatible with older versions of ES or that message
> formats are compatible across major versions at all.
>
>
> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
> user's feedback about it? I cross posted on the user mailing list with
> no luck so far. Personally I would be in favor of dropping the
> connector. Worst case scenario users still have the possibility of
> building the connector themselves from source with just bumping the
> flink's versions. As far as I can tell there were no changes to the code
> base for quite some time.
>
> Best,
>
> Dawid
>
> On 11/02/2020 10:46, Chesnay Schepler wrote:
> > I suppose the downside in an HTTP ES sink is that you don't get _any_
> > form of high-level API from ES, and we'd have to manually build an
> > HTTP request that matches the ES format. Of course you also lose any
> > client-side verification that the clients did, if there is any (but I
> > guess the API itself prevented certain errors).
> >
> > On 11/02/2020 09:32, Stephan Ewen wrote:
> >> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
> >> for that one).
> >>
> >> @Itamar - if you would be interested in contributing a "universal" or
> >> "cross version" ES connector, that could be very interesting. Do you
> >> know
> >> if there are known performance issues or feature restrictions with that
> >> approach?
> >> @dawid what do you think about that?
> >>
> >>
> >> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]>
> wrote:
> >>
> >>> 5.x seems to have a lot of users, is the 6.x completely compatible with
> >>> 5.x ~
> >>>
> >>> Best,
> >>> Danny Chan
> >>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
> >>> <[hidden email]>，写道：
> >>>> Hi all,
> >>>>
> >>>> As described in this
> https://issues.apache.org/jira/browse/FLINK-11720
> >>>> ticket our elasticsearch 5.x connector does not work out of the box on
> >>>> some systems and requires a version bump. This also happens for our
> >>>> e2e.
> >>>> We cannot bump the version in es 5.x connector, because 5.x connector
> >>>> shares a common class with 2.x that uses an API that was replaced
> >>>> in 5.2.
> >>>>
> >>>> Both versions are already long eol:
> https://www.elastic.co/support/eol
> >>>>
> >>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to
> >>>> drop
> >>>> both of them, I would strongly suggest dropping at least 2.x connector
> >>>> and update the 5.x line to a working es client module.
> >>>>
> >>>> What do you think? Should we drop both versions? Drop only the 2.x
> >>>> connector? Or keep them both?
> >>>>
> >>>> Best,
> >>>>
> >>>> Dawid
> >>>>
> >>>>
> >
>
>

[Aljoscha Krettek-2]

Wouldn't removing the ES 2.x connector be enough because we can then
update the ES 5.x connector? It seems there are some users that still
want to use that one.

Best,
Aljoscha

On 18.02.20 10:42, Robert Metzger wrote:

> The ES5 connector is causing some problems on the CI system. It would be
> nice if we could make a decision here soon. I don't want to invest time
> into fixing it, if we are going to remove it.
>
> I'm still in favor of removing it. If we see that there's demand for the
> 5.x connector after the 1.11 release, somebody can take the source and
> contribute it to Apache Bahir or a GitHub account and then posts it to
> flink-packages.org.
>
>
> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz <[hidden email]>
> wrote:
>
>> Sorry for late reply,
>>
>> @all I think there is a general consensus that we want to drop ES 2.x
>> support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
>> track it.
>>
>>
>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
>> Client. ES promises to maintain compatibility of it with any newer minor
>> version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
>> etc.
>>
>> ES provides also a low level rest client which does not include any
>> direct es dependencies and can work with any version of ES. It does not
>> provide any marshalling unmarshalling or higher level features as
>> Chesnay said.
>>
>> Correct me if I am wrong @Itamar but your HTTP client is a simplified
>> version of the ES's high level rest client with a subset of its
>> features. I think it will still have the same problems as ES's High
>> Level Rest Client's because ES does not guarantee that newer message
>> formats will be compatible with older versions of ES or that message
>> formats are compatible across major versions at all.
>>
>>
>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
>> user's feedback about it? I cross posted on the user mailing list with
>> no luck so far. Personally I would be in favor of dropping the
>> connector. Worst case scenario users still have the possibility of
>> building the connector themselves from source with just bumping the
>> flink's versions. As far as I can tell there were no changes to the code
>> base for quite some time.
>>
>> Best,
>>
>> Dawid
>>
>> On 11/02/2020 10:46, Chesnay Schepler wrote:
>>> I suppose the downside in an HTTP ES sink is that you don't get _any_
>>> form of high-level API from ES, and we'd have to manually build an
>>> HTTP request that matches the ES format. Of course you also lose any
>>> client-side verification that the clients did, if there is any (but I
>>> guess the API itself prevented certain errors).
>>>
>>> On 11/02/2020 09:32, Stephan Ewen wrote:
>>>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
>>>> for that one).
>>>>
>>>> @Itamar - if you would be interested in contributing a "universal" or
>>>> "cross version" ES connector, that could be very interesting. Do you
>>>> know
>>>> if there are known performance issues or feature restrictions with that
>>>> approach?
>>>> @dawid what do you think about that?
>>>>
>>>>
>>>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]>
>> wrote:
>>>>
>>>>> 5.x seems to have a lot of users, is the 6.x completely compatible with
>>>>> 5.x ~
>>>>>
>>>>> Best,
>>>>> Danny Chan
>>>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
>>>>> <[hidden email]>，写道：
>>>>>> Hi all,
>>>>>>
>>>>>> As described in this
>> https://issues.apache.org/jira/browse/FLINK-11720
>>>>>> ticket our elasticsearch 5.x connector does not work out of the box on
>>>>>> some systems and requires a version bump. This also happens for our
>>>>>> e2e.
>>>>>> We cannot bump the version in es 5.x connector, because 5.x connector
>>>>>> shares a common class with 2.x that uses an API that was replaced
>>>>>> in 5.2.
>>>>>>
>>>>>> Both versions are already long eol:
>> https://www.elastic.co/support/eol
>>>>>>
>>>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to
>>>>>> drop
>>>>>> both of them, I would strongly suggest dropping at least 2.x connector
>>>>>> and update the 5.x line to a working es client module.
>>>>>>
>>>>>> What do you think? Should we drop both versions? Drop only the 2.x
>>>>>> connector? Or keep them both?
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>>
>>>
>>
>>
>

[Chesnay Schepler-3]

Since one of the reasons for dropping ES2 was that it blocks some
critical updates for the ES5 connector I'd prefer to keep ES5 around for
1.11, and revisit this discussion for 1.12 .

On 18/02/2020 13:03, Aljoscha Krettek wrote:

> Wouldn't removing the ES 2.x connector be enough because we can then
> update the ES 5.x connector? It seems there are some users that still
> want to use that one.
>
> Best,
> Aljoscha
>
> On 18.02.20 10:42, Robert Metzger wrote:
>> The ES5 connector is causing some problems on the CI system. It would be
>> nice if we could make a decision here soon. I don't want to invest time
>> into fixing it, if we are going to remove it.
>>
>> I'm still in favor of removing it. If we see that there's demand for the
>> 5.x connector after the 1.11 release, somebody can take the source and
>> contribute it to Apache Bahir or a GitHub account and then posts it to
>> flink-packages.org.
>>
>>
>> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz
>> <[hidden email]>
>> wrote:
>>
>>> Sorry for late reply,
>>>
>>> @all I think there is a general consensus that we want to drop ES 2.x
>>> support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
>>> track it.
>>>
>>>
>>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
>>> Client. ES promises to maintain compatibility of it with any newer
>>> minor
>>> version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
>>> etc.
>>>
>>> ES provides also a low level rest client which does not include any
>>> direct es dependencies and can work with any version of ES. It does not
>>> provide any marshalling unmarshalling or higher level features as
>>> Chesnay said.
>>>
>>> Correct me if I am wrong @Itamar but your HTTP client is a simplified
>>> version of the ES's high level rest client with a subset of its
>>> features. I think it will still have the same problems as ES's High
>>> Level Rest Client's because ES does not guarantee that newer message
>>> formats will be compatible with older versions of ES or that message
>>> formats are compatible across major versions at all.
>>>
>>>
>>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
>>> user's feedback about it? I cross posted on the user mailing list with
>>> no luck so far. Personally I would be in favor of dropping the
>>> connector. Worst case scenario users still have the possibility of
>>> building the connector themselves from source with just bumping the
>>> flink's versions. As far as I can tell there were no changes to the
>>> code
>>> base for quite some time.
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> On 11/02/2020 10:46, Chesnay Schepler wrote:
>>>> I suppose the downside in an HTTP ES sink is that you don't get _any_
>>>> form of high-level API from ES, and we'd have to manually build an
>>>> HTTP request that matches the ES format. Of course you also lose any
>>>> client-side verification that the clients did, if there is any (but I
>>>> guess the API itself prevented certain errors).
>>>>
>>>> On 11/02/2020 09:32, Stephan Ewen wrote:
>>>>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user
>>>>> input
>>>>> for that one).
>>>>>
>>>>> @Itamar - if you would be interested in contributing a "universal" or
>>>>> "cross version" ES connector, that could be very interesting. Do you
>>>>> know
>>>>> if there are known performance issues or feature restrictions with
>>>>> that
>>>>> approach?
>>>>> @dawid what do you think about that?
>>>>>
>>>>>
>>>>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]>
>>> wrote:
>>>>>
>>>>>> 5.x seems to have a lot of users, is the 6.x completely
>>>>>> compatible with
>>>>>> 5.x ~
>>>>>>
>>>>>> Best,
>>>>>> Danny Chan
>>>>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
>>>>>> <[hidden email]>，写道：
>>>>>>> Hi all,
>>>>>>>
>>>>>>> As described in this
>>> https://issues.apache.org/jira/browse/FLINK-11720
>>>>>>> ticket our elasticsearch 5.x connector does not work out of the
>>>>>>> box on
>>>>>>> some systems and requires a version bump. This also happens for our
>>>>>>> e2e.
>>>>>>> We cannot bump the version in es 5.x connector, because 5.x
>>>>>>> connector
>>>>>>> shares a common class with 2.x that uses an API that was replaced
>>>>>>> in 5.2.
>>>>>>>
>>>>>>> Both versions are already long eol:
>>> https://www.elastic.co/support/eol
>>>>>>>
>>>>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to
>>>>>>> drop
>>>>>>> both of them, I would strongly suggest dropping at least 2.x
>>>>>>> connector
>>>>>>> and update the 5.x line to a working es client module.
>>>>>>>
>>>>>>> What do you think? Should we drop both versions? Drop only the 2.x
>>>>>>> connector? Or keep them both?
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>>
>>>>
>>>
>>>
>>
>

[Till Rohrmann]

+1 for removing ES2 and updating ES5 in 1.11.

+1 for revisiting the removal of ES5 for 1.12.

Cheers,
Till

On Tue, Feb 18, 2020 at 1:28 PM Chesnay Schepler <[hidden email]> wrote:

> Since one of the reasons for dropping ES2 was that it blocks some
> critical updates for the ES5 connector I'd prefer to keep ES5 around for
> 1.11, and revisit this discussion for 1.12 .
>
> On 18/02/2020 13:03, Aljoscha Krettek wrote:
> > Wouldn't removing the ES 2.x connector be enough because we can then
> > update the ES 5.x connector? It seems there are some users that still
> > want to use that one.
> >
> > Best,
> > Aljoscha
> >
> > On 18.02.20 10:42, Robert Metzger wrote:
> >> The ES5 connector is causing some problems on the CI system. It would be
> >> nice if we could make a decision here soon. I don't want to invest time
> >> into fixing it, if we are going to remove it.
> >>
> >> I'm still in favor of removing it. If we see that there's demand for the
> >> 5.x connector after the 1.11 release, somebody can take the source and
> >> contribute it to Apache Bahir or a GitHub account and then posts it to
> >> flink-packages.org.
> >>
> >>
> >> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz
> >> <[hidden email]>
> >> wrote:
> >>
> >>> Sorry for late reply,
> >>>
> >>> @all I think there is a general consensus that we want to drop ES 2.x
> >>> support. I created https://issues.apache.org/jira/browse/FLINK-16046
> to
> >>> track it.
> >>>
> >>>
> >>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
> >>> Client. ES promises to maintain compatibility of it with any newer
> >>> minor
> >>> version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
> >>> etc.
> >>>
> >>> ES provides also a low level rest client which does not include any
> >>> direct es dependencies and can work with any version of ES. It does not
> >>> provide any marshalling unmarshalling or higher level features as
> >>> Chesnay said.
> >>>
> >>> Correct me if I am wrong @Itamar but your HTTP client is a simplified
> >>> version of the ES's high level rest client with a subset of its
> >>> features. I think it will still have the same problems as ES's High
> >>> Level Rest Client's because ES does not guarantee that newer message
> >>> formats will be compatible with older versions of ES or that message
> >>> formats are compatible across major versions at all.
> >>>
> >>>
> >>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
> >>> user's feedback about it? I cross posted on the user mailing list with
> >>> no luck so far. Personally I would be in favor of dropping the
> >>> connector. Worst case scenario users still have the possibility of
> >>> building the connector themselves from source with just bumping the
> >>> flink's versions. As far as I can tell there were no changes to the
> >>> code
> >>> base for quite some time.
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>>
> >>> On 11/02/2020 10:46, Chesnay Schepler wrote:
> >>>> I suppose the downside in an HTTP ES sink is that you don't get _any_
> >>>> form of high-level API from ES, and we'd have to manually build an
> >>>> HTTP request that matches the ES format. Of course you also lose any
> >>>> client-side verification that the clients did, if there is any (but I
> >>>> guess the API itself prevented certain errors).
> >>>>
> >>>> On 11/02/2020 09:32, Stephan Ewen wrote:
> >>>>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user
> >>>>> input
> >>>>> for that one).
> >>>>>
> >>>>> @Itamar - if you would be interested in contributing a "universal" or
> >>>>> "cross version" ES connector, that could be very interesting. Do you
> >>>>> know
> >>>>> if there are known performance issues or feature restrictions with
> >>>>> that
> >>>>> approach?
> >>>>> @dawid what do you think about that?
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]>
> >>> wrote:
> >>>>>
> >>>>>> 5.x seems to have a lot of users, is the 6.x completely
> >>>>>> compatible with
> >>>>>> 5.x ~
> >>>>>>
> >>>>>> Best,
> >>>>>> Danny Chan
> >>>>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
> >>>>>> <[hidden email]>，写道：
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> As described in this
> >>> https://issues.apache.org/jira/browse/FLINK-11720
> >>>>>>> ticket our elasticsearch 5.x connector does not work out of the
> >>>>>>> box on
> >>>>>>> some systems and requires a version bump. This also happens for our
> >>>>>>> e2e.
> >>>>>>> We cannot bump the version in es 5.x connector, because 5.x
> >>>>>>> connector
> >>>>>>> shares a common class with 2.x that uses an API that was replaced
> >>>>>>> in 5.2.
> >>>>>>>
> >>>>>>> Both versions are already long eol:
> >>> https://www.elastic.co/support/eol
> >>>>>>>
> >>>>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to
> >>>>>>> drop
> >>>>>>> both of them, I would strongly suggest dropping at least 2.x
> >>>>>>> connector
> >>>>>>> and update the 5.x line to a working es client module.
> >>>>>>>
> >>>>>>> What do you think? Should we drop both versions? Drop only the 2.x
> >>>>>>> connector? Or keep them both?
> >>>>>>>
> >>>>>>> Best,
> >>>>>>>
> >>>>>>> Dawid
> >>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>>
> >>
> >
>
>

[Robert Metzger]

Thank you all.

I have proposed a PR to upgrade the ES5 connector:
https://github.com/apache/flink/pull/11162

On Wed, Feb 19, 2020 at 3:02 PM Till Rohrmann <[hidden email]> wrote:

> +1 for removing ES2 and updating ES5 in 1.11.
>
> +1 for revisiting the removal of ES5 for 1.12.
>
> Cheers,
> Till
>
> On Tue, Feb 18, 2020 at 1:28 PM Chesnay Schepler <[hidden email]>
> wrote:
>
> > Since one of the reasons for dropping ES2 was that it blocks some
> > critical updates for the ES5 connector I'd prefer to keep ES5 around for
> > 1.11, and revisit this discussion for 1.12 .
> >
> > On 18/02/2020 13:03, Aljoscha Krettek wrote:
> > > Wouldn't removing the ES 2.x connector be enough because we can then
> > > update the ES 5.x connector? It seems there are some users that still
> > > want to use that one.
> > >
> > > Best,
> > > Aljoscha
> > >
> > > On 18.02.20 10:42, Robert Metzger wrote:
> > >> The ES5 connector is causing some problems on the CI system. It would
> be
> > >> nice if we could make a decision here soon. I don't want to invest
> time
> > >> into fixing it, if we are going to remove it.
> > >>
> > >> I'm still in favor of removing it. If we see that there's demand for
> the
> > >> 5.x connector after the 1.11 release, somebody can take the source and
> > >> contribute it to Apache Bahir or a GitHub account and then posts it to
> > >> flink-packages.org.
> > >>
> > >>
> > >> On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz
> > >> <[hidden email]>
> > >> wrote:
> > >>
> > >>> Sorry for late reply,
> > >>>
> > >>> @all I think there is a general consensus that we want to drop ES 2.x
> > >>> support. I created https://issues.apache.org/jira/browse/FLINK-16046
> > to
> > >>> track it.
> > >>>
> > >>>
> > >>> @Stephan @Chesnay @Itamar In our connectors we use Java High Level
> Rest
> > >>> Client. ES promises to maintain compatibility of it with any newer
> > >>> minor
> > >>> version of ES. So if we have 6.1 client we can use it with any 6.2,
> 6.3
> > >>> etc.
> > >>>
> > >>> ES provides also a low level rest client which does not include any
> > >>> direct es dependencies and can work with any version of ES. It does
> not
> > >>> provide any marshalling unmarshalling or higher level features as
> > >>> Chesnay said.
> > >>>
> > >>> Correct me if I am wrong @Itamar but your HTTP client is a simplified
> > >>> version of the ES's high level rest client with a subset of its
> > >>> features. I think it will still have the same problems as ES's High
> > >>> Level Rest Client's because ES does not guarantee that newer message
> > >>> formats will be compatible with older versions of ES or that message
> > >>> formats are compatible across major versions at all.
> > >>>
> > >>>
> > >>> @Stephan @Danny As for the 5.x connector. Any ideas how can we get
> > >>> user's feedback about it? I cross posted on the user mailing list
> with
> > >>> no luck so far. Personally I would be in favor of dropping the
> > >>> connector. Worst case scenario users still have the possibility of
> > >>> building the connector themselves from source with just bumping the
> > >>> flink's versions. As far as I can tell there were no changes to the
> > >>> code
> > >>> base for quite some time.
> > >>>
> > >>> Best,
> > >>>
> > >>> Dawid
> > >>>
> > >>> On 11/02/2020 10:46, Chesnay Schepler wrote:
> > >>>> I suppose the downside in an HTTP ES sink is that you don't get
> _any_
> > >>>> form of high-level API from ES, and we'd have to manually build an
> > >>>> HTTP request that matches the ES format. Of course you also lose any
> > >>>> client-side verification that the clients did, if there is any (but
> I
> > >>>> guess the API itself prevented certain errors).
> > >>>>
> > >>>> On 11/02/2020 09:32, Stephan Ewen wrote:
> > >>>>> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user
> > >>>>> input
> > >>>>> for that one).
> > >>>>>
> > >>>>> @Itamar - if you would be interested in contributing a "universal"
> or
> > >>>>> "cross version" ES connector, that could be very interesting. Do
> you
> > >>>>> know
> > >>>>> if there are known performance issues or feature restrictions with
> > >>>>> that
> > >>>>> approach?
> > >>>>> @dawid what do you think about that?
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <[hidden email]>
> > >>> wrote:
> > >>>>>
> > >>>>>> 5.x seems to have a lot of users, is the 6.x completely
> > >>>>>> compatible with
> > >>>>>> 5.x ~
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Danny Chan
> > >>>>>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
> > >>>>>> <[hidden email]>，写道：
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> As described in this
> > >>> https://issues.apache.org/jira/browse/FLINK-11720
> > >>>>>>> ticket our elasticsearch 5.x connector does not work out of the
> > >>>>>>> box on
> > >>>>>>> some systems and requires a version bump. This also happens for
> our
> > >>>>>>> e2e.
> > >>>>>>> We cannot bump the version in es 5.x connector, because 5.x
> > >>>>>>> connector
> > >>>>>>> shares a common class with 2.x that uses an API that was replaced
> > >>>>>>> in 5.2.
> > >>>>>>>
> > >>>>>>> Both versions are already long eol:
> > >>> https://www.elastic.co/support/eol
> > >>>>>>>
> > >>>>>>> I suggest to drop both connectors 5.x and 2.x. If it is too much
> to
> > >>>>>>> drop
> > >>>>>>> both of them, I would strongly suggest dropping at least 2.x
> > >>>>>>> connector
> > >>>>>>> and update the 5.x line to a working es client module.
> > >>>>>>>
> > >>>>>>> What do you think? Should we drop both versions? Drop only the
> 2.x
> > >>>>>>> connector? Or keep them both?
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>>
> > >>>>>>> Dawid
> > >>>>>>>
> > >>>>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
> >
>