Elasticsearch connector

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch connector

Premkumar, Ramprabhu
Hi Team,

Could you please let me know if the elasticsearch connector from Flink
1.3.2 support elasticsearch 5.5.1. I am not able to find the add method in
RequestIndexer interface using this connector.

Dependency used:

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-elasticsearch5_2.10</artifactId>
    <version>1.3.2</version>
</dependency>

Thanks
Ramprabhu Premkumar
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch connector

Fabian Hueske-2
Hi,

that's correct. Elasticsearch changed its API from 5.1 to 5.2.
There is a JIRA issue for the problem [1] and a pull request [2].

It would be great if you could help to resolve the problem by verifying the
PR.

Thanks, Fabian

[1] https://issues.apache.org/jira/browse/FLINK-7386
[2] https://github.com/apache/flink/pull/4675

2017-09-15 19:05 GMT+02:00 Premkumar, Ramprabhu <
[hidden email]>:

> Hi Team,
>
> Could you please let me know if the elasticsearch connector from Flink
> 1.3.2 support elasticsearch 5.5.1. I am not able to find the add method in
> RequestIndexer interface using this connector.
>
> Dependency used:
>
> <dependency>
>     <groupId>org.apache.flink</groupId>
>     <artifactId>flink-connector-elasticsearch5_2.10</artifactId>
>     <version>1.3.2</version>
> </dependency>
>
> Thanks
> Ramprabhu Premkumar
>
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch connector

Premkumar, Ramprabhu
Hi,

Receiving this exception with the 1.3.2 dependencies for elastic search
connector.

java.lang.NoSuchMethodError:
org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/elasticsearch/action/ActionRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor;
at
org.apache.flink.streaming.connectors.elasticsearch.BulkProcessorIndexer.add(BulkProcessorIndexer.java:52)


But the I see this in the code base in BulkProcessor Class

    public BulkProcessor add(IndexRequest request) {
        return add((DocWriteRequest) request);
    }

Could you please help in letting me know why is this exception being thrown
despite the method definition existing ?

Dependencies used :

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch5_2.11</artifactId>
<version>1.3.2</version>
<exclusions>

<exclusion>

<groupId>org.elasticsearch.client</groupId>

<artifactId>transport</artifactId>

</exclusion>

</exclusions>
</dependency>

<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>x-pack-transport</artifactId>
<version>5.5.1</version>
</dependency>


Thanks


On Tue, Sep 19, 2017 at 3:42 AM, Fabian Hueske <[hidden email]> wrote:

> Hi,
>
> that's correct. Elasticsearch changed its API from 5.1 to 5.2.
> There is a JIRA issue for the problem [1] and a pull request [2].
>
> It would be great if you could help to resolve the problem by verifying the
> PR.
>
> Thanks, Fabian
>
> [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_FLINK-2D7386&d=DwIBaQ&c=
> udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=B-nyt4heL8e6bDgEOtGErOdVe8e-
> DMB2iPYpRfqRe-A&m=TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=
> cq_sXWcRAEv-wXNoxdNa-IMc_-Kf4rCHK1xsW4Dn8uk&e=
> [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_flink_pull_4675&d=DwIBaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LR
> xpb6__0PomBTQ&r=B-nyt4heL8e6bDgEOtGErOdVe8e-DMB2iPYpRfqRe-A&m=
> TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=X13U81ugZMK53oPA-
> Gk3CLvP0rzFr2Xi2ix1UgoqzBc&e=
>
> 2017-09-15 19:05 GMT+02:00 Premkumar, Ramprabhu <
> [hidden email]>:
>
> > Hi Team,
> >
> > Could you please let me know if the elasticsearch connector from Flink
> > 1.3.2 support elasticsearch 5.5.1. I am not able to find the add method
> in
> > RequestIndexer interface using this connector.
> >
> > Dependency used:
> >
> > <dependency>
> >     <groupId>org.apache.flink</groupId>
> >     <artifactId>flink-connector-elasticsearch5_2.10</artifactId>
> >     <version>1.3.2</version>
> > </dependency>
> >
> > Thanks
> > Ramprabhu Premkumar
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch connector

Till Rohrmann
Hi,

usually such an error indicates that you have a different version of
Elasticsearch in your classpath than the one used at compile time.

Please make sure that you don't have accidentally an older version of
Elasticsearch in your classpath. You can see the complete classpath at the
beginning of the logs.

If this is not the case, then please check that the right version is
included in the Flink jar by checking whether the above-mentioned method is
contained in the class file via `jad`.

Cheers,
Till

On Tue, Sep 26, 2017 at 12:37 AM, Premkumar, Ramprabhu <
[hidden email]> wrote:

> Hi,
>
> Receiving this exception with the 1.3.2 dependencies for elastic search
> connector.
>
> java.lang.NoSuchMethodError:
> org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/elasticsearch/action/
> ActionRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor;
> at
> org.apache.flink.streaming.connectors.elasticsearch.
> BulkProcessorIndexer.add(BulkProcessorIndexer.java:52)
>
>
> But the I see this in the code base in BulkProcessor Class
>
>     public BulkProcessor add(IndexRequest request) {
>         return add((DocWriteRequest) request);
>     }
>
> Could you please help in letting me know why is this exception being thrown
> despite the method definition existing ?
>
> Dependencies used :
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-connector-elasticsearch5_2.11</artifactId>
> <version>1.3.2</version>
> <exclusions>
>
> <exclusion>
>
> <groupId>org.elasticsearch.client</groupId>
>
> <artifactId>transport</artifactId>
>
> </exclusion>
>
> </exclusions>
> </dependency>
>
> <dependency>
> <groupId>org.elasticsearch.client</groupId>
> <artifactId>x-pack-transport</artifactId>
> <version>5.5.1</version>
> </dependency>
>
>
> Thanks
>
>
> On Tue, Sep 19, 2017 at 3:42 AM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi,
> >
> > that's correct. Elasticsearch changed its API from 5.1 to 5.2.
> > There is a JIRA issue for the problem [1] and a pull request [2].
> >
> > It would be great if you could help to resolve the problem by verifying
> the
> > PR.
> >
> > Thanks, Fabian
> >
> > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > apache.org_jira_browse_FLINK-2D7386&d=DwIBaQ&c=
> > udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=B-
> nyt4heL8e6bDgEOtGErOdVe8e-
> > DMB2iPYpRfqRe-A&m=TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=
> > cq_sXWcRAEv-wXNoxdNa-IMc_-Kf4rCHK1xsW4Dn8uk&e=
> > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> > com_apache_flink_pull_4675&d=DwIBaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LR
> > xpb6__0PomBTQ&r=B-nyt4heL8e6bDgEOtGErOdVe8e-DMB2iPYpRfqRe-A&m=
> > TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=X13U81ugZMK53oPA-
> > Gk3CLvP0rzFr2Xi2ix1UgoqzBc&e=
> >
> > 2017-09-15 19:05 GMT+02:00 Premkumar, Ramprabhu <
> > [hidden email]>:
> >
> > > Hi Team,
> > >
> > > Could you please let me know if the elasticsearch connector from Flink
> > > 1.3.2 support elasticsearch 5.5.1. I am not able to find the add method
> > in
> > > RequestIndexer interface using this connector.
> > >
> > > Dependency used:
> > >
> > > <dependency>
> > >     <groupId>org.apache.flink</groupId>
> > >     <artifactId>flink-connector-elasticsearch5_2.10</artifactId>
> > >     <version>1.3.2</version>
> > > </dependency>
> > >
> > > Thanks
> > > Ramprabhu Premkumar
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [E] Re: Elasticsearch connector

Premkumar, Ramprabhu
Thank you for the response!

It turns out actually that elastic 5.5.1 was throwing that exception but
when I used the elastic 5.1.1 it works fine.

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>5.1.1</version>
</dependency>

Like Fabian Hueske mentioned previously it is not working with 5.5.1 and
probably elastic versions above 5.2

One item which I am still not able to understand is I found the `add`
method definition in the BulkProcessor class.

    public BulkProcessor add(IndexRequest request) {
        return add((DocWriteRequest) request);
    }

Thanks



On Tue, Sep 26, 2017 at 2:59 AM, Till Rohrmann <[hidden email]> wrote:

> Hi,
>
> usually such an error indicates that you have a different version of
> Elasticsearch in your classpath than the one used at compile time.
>
> Please make sure that you don't have accidentally an older version of
> Elasticsearch in your classpath. You can see the complete classpath at the
> beginning of the logs.
>
> If this is not the case, then please check that the right version is
> included in the Flink jar by checking whether the above-mentioned method is
> contained in the class file via `jad`.
>
> Cheers,
> Till
>
> On Tue, Sep 26, 2017 at 12:37 AM, Premkumar, Ramprabhu <
> [hidden email]> wrote:
>
> > Hi,
> >
> > Receiving this exception with the 1.3.2 dependencies for elastic search
> > connector.
> >
> > java.lang.NoSuchMethodError:
> > org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/
> elasticsearch/action/
> > ActionRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor;
> > at
> > org.apache.flink.streaming.connectors.elasticsearch.
> > BulkProcessorIndexer.add(BulkProcessorIndexer.java:52)
> >
> >
> > But the I see this in the code base in BulkProcessor Class
> >
> >     public BulkProcessor add(IndexRequest request) {
> >         return add((DocWriteRequest) request);
> >     }
> >
> > Could you please help in letting me know why is this exception being
> thrown
> > despite the method definition existing ?
> >
> > Dependencies used :
> >
> > <dependency>
> > <groupId>org.apache.flink</groupId>
> > <artifactId>flink-connector-elasticsearch5_2.11</artifactId>
> > <version>1.3.2</version>
> > <exclusions>
> >
> > <exclusion>
> >
> > <groupId>org.elasticsearch.client</groupId>
> >
> > <artifactId>transport</artifactId>
> >
> > </exclusion>
> >
> > </exclusions>
> > </dependency>
> >
> > <dependency>
> > <groupId>org.elasticsearch.client</groupId>
> > <artifactId>x-pack-transport</artifactId>
> > <version>5.5.1</version>
> > </dependency>
> >
> >
> > Thanks
> >
> >
> > On Tue, Sep 19, 2017 at 3:42 AM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi,
> > >
> > > that's correct. Elasticsearch changed its API from 5.1 to 5.2.
> > > There is a JIRA issue for the problem [1] and a pull request [2].
> > >
> > > It would be great if you could help to resolve the problem by verifying
> > the
> > > PR.
> > >
> > > Thanks, Fabian
> > >
> > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > > apache.org_jira_browse_FLINK-2D7386&d=DwIBaQ&c=
> > > udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=B-
> > nyt4heL8e6bDgEOtGErOdVe8e-
> > > DMB2iPYpRfqRe-A&m=TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=
> > > cq_sXWcRAEv-wXNoxdNa-IMc_-Kf4rCHK1xsW4Dn8uk&e=
> > > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> > > com_apache_flink_pull_4675&d=DwIBaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LR
> > > xpb6__0PomBTQ&r=B-nyt4heL8e6bDgEOtGErOdVe8e-DMB2iPYpRfqRe-A&m=
> > > TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=X13U81ugZMK53oPA-
> > > Gk3CLvP0rzFr2Xi2ix1UgoqzBc&e=
> > >
> > > 2017-09-15 19:05 GMT+02:00 Premkumar, Ramprabhu <
> > > [hidden email]>:
> > >
> > > > Hi Team,
> > > >
> > > > Could you please let me know if the elasticsearch connector from
> Flink
> > > > 1.3.2 support elasticsearch 5.5.1. I am not able to find the add
> method
> > > in
> > > > RequestIndexer interface using this connector.
> > > >
> > > > Dependency used:
> > > >
> > > > <dependency>
> > > >     <groupId>org.apache.flink</groupId>
> > > >     <artifactId>flink-connector-elasticsearch5_2.10</artifactId>
> > > >     <version>1.3.2</version>
> > > > </dependency>
> > > >
> > > > Thanks
> > > > Ramprabhu Premkumar
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [E] Re: Elasticsearch connector

Till Rohrmann
The problem are the different signatures. The old version seem to take
an `ActionRequest`
whereas the old version requires a `IndexRequest`.

Cheers,
Till

On Tue, Sep 26, 2017 at 10:09 PM, Premkumar, Ramprabhu <
[hidden email]> wrote:

> Thank you for the response!
>
> It turns out actually that elastic 5.5.1 was throwing that exception but
> when I used the elastic 5.1.1 it works fine.
>
> <dependency>
>     <groupId>org.elasticsearch</groupId>
>     <artifactId>elasticsearch</artifactId>
>     <version>5.1.1</version>
> </dependency>
>
> Like Fabian Hueske mentioned previously it is not working with 5.5.1 and
> probably elastic versions above 5.2
>
> One item which I am still not able to understand is I found the `add`
> method definition in the BulkProcessor class.
>
>     public BulkProcessor add(IndexRequest request) {
>         return add((DocWriteRequest) request);
>     }
>
> Thanks
>
>
>
> On Tue, Sep 26, 2017 at 2:59 AM, Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi,
> >
> > usually such an error indicates that you have a different version of
> > Elasticsearch in your classpath than the one used at compile time.
> >
> > Please make sure that you don't have accidentally an older version of
> > Elasticsearch in your classpath. You can see the complete classpath at
> the
> > beginning of the logs.
> >
> > If this is not the case, then please check that the right version is
> > included in the Flink jar by checking whether the above-mentioned method
> is
> > contained in the class file via `jad`.
> >
> > Cheers,
> > Till
> >
> > On Tue, Sep 26, 2017 at 12:37 AM, Premkumar, Ramprabhu <
> > [hidden email]> wrote:
> >
> > > Hi,
> > >
> > > Receiving this exception with the 1.3.2 dependencies for elastic search
> > > connector.
> > >
> > > java.lang.NoSuchMethodError:
> > > org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/
> > elasticsearch/action/
> > > ActionRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor;
> > > at
> > > org.apache.flink.streaming.connectors.elasticsearch.
> > > BulkProcessorIndexer.add(BulkProcessorIndexer.java:52)
> > >
> > >
> > > But the I see this in the code base in BulkProcessor Class
> > >
> > >     public BulkProcessor add(IndexRequest request) {
> > >         return add((DocWriteRequest) request);
> > >     }
> > >
> > > Could you please help in letting me know why is this exception being
> > thrown
> > > despite the method definition existing ?
> > >
> > > Dependencies used :
> > >
> > > <dependency>
> > > <groupId>org.apache.flink</groupId>
> > > <artifactId>flink-connector-elasticsearch5_2.11</artifactId>
> > > <version>1.3.2</version>
> > > <exclusions>
> > >
> > > <exclusion>
> > >
> > > <groupId>org.elasticsearch.client</groupId>
> > >
> > > <artifactId>transport</artifactId>
> > >
> > > </exclusion>
> > >
> > > </exclusions>
> > > </dependency>
> > >
> > > <dependency>
> > > <groupId>org.elasticsearch.client</groupId>
> > > <artifactId>x-pack-transport</artifactId>
> > > <version>5.5.1</version>
> > > </dependency>
> > >
> > >
> > > Thanks
> > >
> > >
> > > On Tue, Sep 19, 2017 at 3:42 AM, Fabian Hueske <[hidden email]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > that's correct. Elasticsearch changed its API from 5.1 to 5.2.
> > > > There is a JIRA issue for the problem [1] and a pull request [2].
> > > >
> > > > It would be great if you could help to resolve the problem by
> verifying
> > > the
> > > > PR.
> > > >
> > > > Thanks, Fabian
> > > >
> > > > [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > > > apache.org_jira_browse_FLINK-2D7386&d=DwIBaQ&c=
> > > > udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=B-
> > > nyt4heL8e6bDgEOtGErOdVe8e-
> > > > DMB2iPYpRfqRe-A&m=TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=
> > > > cq_sXWcRAEv-wXNoxdNa-IMc_-Kf4rCHK1xsW4Dn8uk&e=
> > > > [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> > > > com_apache_flink_pull_4675&d=DwIBaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LR
> > > > xpb6__0PomBTQ&r=B-nyt4heL8e6bDgEOtGErOdVe8e-DMB2iPYpRfqRe-A&m=
> > > > TAQ1vm3DOiBGJpe-EfbNyMVcz8zx5EYiGxx4kX4-BRo&s=X13U81ugZMK53oPA-
> > > > Gk3CLvP0rzFr2Xi2ix1UgoqzBc&e=
> > > >
> > > > 2017-09-15 19:05 GMT+02:00 Premkumar, Ramprabhu <
> > > > [hidden email]>:
> > > >
> > > > > Hi Team,
> > > > >
> > > > > Could you please let me know if the elasticsearch connector from
> > Flink
> > > > > 1.3.2 support elasticsearch 5.5.1. I am not able to find the add
> > method
> > > > in
> > > > > RequestIndexer interface using this connector.
> > > > >
> > > > > Dependency used:
> > > > >
> > > > > <dependency>
> > > > >     <groupId>org.apache.flink</groupId>
> > > > >     <artifactId>flink-connector-elasticsearch5_2.10</artifactId>
> > > > >     <version>1.3.2</version>
> > > > > </dependency>
> > > > >
> > > > > Thanks
> > > > > Ramprabhu Premkumar
> > > > >
> > > >
> > >
> >
>