Apache flink with Elastic Search

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache flink with Elastic Search

santosh_rajaguru
Hi all,

Is there any possibility to query Elastic search using flink?


Thanks and Regards,
Santosh
Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

Sachin Goel
Hi Santosh
There is an Elastic search connector under streaming connectors. Can you
check if that satisfies your needs?
Aljoscha will be able to provide more details in case you need them.

Cheers!
Sachin

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685

On Tue, Oct 13, 2015 at 6:38 PM, santosh_rajaguru <[hidden email]> wrote:

> Hi all,
>
> Is there any possibility to query Elastic search using flink?
>
>
> Thanks and Regards,
> Santosh
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

Aljoscha Krettek-2
Hi,
the Elasticsearch connector can only be used for writing right now. If there is need we could also think about adding a connector for reading, though.

Cheers,
Aljoscha

> On 13 Oct 2015, at 16:01, Sachin Goel <[hidden email]> wrote:
>
> Hi Santosh
> There is an Elastic search connector under streaming connectors. Can you
> check if that satisfies your needs?
> Aljoscha will be able to provide more details in case you need them.
>
> Cheers!
> Sachin
>
> -- Sachin Goel
> Computer Science, IIT Delhi
> m. +91-9871457685
>
> On Tue, Oct 13, 2015 at 6:38 PM, santosh_rajaguru <[hidden email]> wrote:
>
>> Hi all,
>>
>> Is there any possibility to query Elastic search using flink?
>>
>>
>> Thanks and Regards,
>> Santosh
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
>> Sent from the Apache Flink Mailing List archive. mailing list archive at
>> Nabble.com.
>>

Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

Flavio Pompermaier
You can always use the Hadoop connector of Elasticsearch ;)
On 14 Oct 2015 08:32, "Aljoscha Krettek" <[hidden email]> wrote:

> Hi,
> the Elasticsearch connector can only be used for writing right now. If
> there is need we could also think about adding a connector for reading,
> though.
>
> Cheers,
> Aljoscha
> > On 13 Oct 2015, at 16:01, Sachin Goel <[hidden email]> wrote:
> >
> > Hi Santosh
> > There is an Elastic search connector under streaming connectors. Can you
> > check if that satisfies your needs?
> > Aljoscha will be able to provide more details in case you need them.
> >
> > Cheers!
> > Sachin
> >
> > -- Sachin Goel
> > Computer Science, IIT Delhi
> > m. +91-9871457685
> >
> > On Tue, Oct 13, 2015 at 6:38 PM, santosh_rajaguru <[hidden email]>
> wrote:
> >
> >> Hi all,
> >>
> >> Is there any possibility to query Elastic search using flink?
> >>
> >>
> >> Thanks and Regards,
> >> Santosh
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435.html
> >> Sent from the Apache Flink Mailing List archive. mailing list archive at
> >> Nabble.com.
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

santosh_rajaguru
Thanks flavio.
Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

Robert Metzger
I think the Hadoop connector of ElasticSearch only works for batch jobs.
In my understanding, elasticsearch also allows "streaming" standing search
queries.

On Wed, Oct 14, 2015 at 10:41 AM, santosh_rajaguru <[hidden email]>
wrote:

> Thanks flavio.
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435p8445.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

Martin Liesenberg
Elasticsearch does not support streaming in the sense, that you subscribe
to a search query and as new documents come in, you will get any matching
your query.
There is a rather long discussion in one of the issues in the ES repository
on GitHub concerning this (active for 4 years now) [1]

It does however provide you with an API to scroll through a large set of
results, so you can retrieve them in chunks which fit your application. [2]
That is supported by the hadoop connector via [3] which is used as far as I
can tell in almost all of the different connectors (hive, cascading, mr,
spark)

Best regards,
Martin

[1] https://github.com/elastic/elasticsearch/issues/1242
[2]
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
[3]
https://github.com/elastic/elasticsearch-hadoop/blob/03c056142a5ab7422b81bb1f519fd67a9581405f/mr/src/main/java/org/elasticsearch/hadoop/rest/ScrollQuery.java

Robert Metzger <[hidden email]> schrieb am Mi., 14. Okt. 2015 um
17:49 Uhr:

> I think the Hadoop connector of ElasticSearch only works for batch jobs.
> In my understanding, elasticsearch also allows "streaming" standing search
> queries.
>
> On Wed, Oct 14, 2015 at 10:41 AM, santosh_rajaguru <[hidden email]>
> wrote:
>
> > Thanks flavio.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-flink-with-Elastic-Search-tp8435p8445.html
> > Sent from the Apache Flink Mailing List archive. mailing list archive at
> > Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Apache flink with Elastic Search

santosh_rajaguru
Thanks Martin for the input.