No support for request PutMappingRequest

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

No support for request PutMappingRequest

Ozan DENİZ
Hi everyone,


We are trying to use elasticsearch (2.x) connector for Flink application. However, we encounter a problem when we try to add mapping to elasticsearch index.


The error message is below when we run the Flink application.


 No support for request [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest]


We are using "putMappingRequest" function for mapping. In Flink, is there any way to add mapping to indexes in elasticsearch?


If not, can we contribute for adding mapping feature to elasticsearch connector?
Reply | Threaded
Open this post in threaded view
|

Re: No support for request PutMappingRequest

Till Rohrmann
Hi Ozan,

I'm not super experienced with Flink's elasticsearch connector, but could
you post the complete stack trace to figure out where the problem comes
from?

I've also pulled in Aljoscha, the original author of the elasticsearch
sink. Maybe he can give you a detailed answer.

Cheers,
Till

On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote:

> Hi everyone,
>
>
> We are trying to use elasticsearch (2.x) connector for Flink application.
> However, we encounter a problem when we try to add mapping to elasticsearch
> index.
>
>
> The error message is below when we run the Flink application.
>
>
>  No support for request [org.elasticsearch.action.
> admin.indices.mapping.put.PutMappingRequest]
>
>
> We are using "putMappingRequest" function for mapping. In Flink, is there
> any way to add mapping to indexes in elasticsearch?
>
>
> If not, can we contribute for adding mapping feature to elasticsearch
> connector?
>
Reply | Threaded
Open this post in threaded view
|

Ynt: No support for request PutMappingRequest

Ozan DENİZ
Hi,


I am sending the error message below;


java.lang.IllegalArgumentException: No support for request [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b]
at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107) ~[elasticsearch-2.3.5.jar:2.3.5]
at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284) ~[elasticsearch-2.3.5.jar:2.3.5]
at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268) ~[elasticsearch-2.3.5.jar:2.3.5]
at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264) ~[elasticsearch-2.3.5.jar:2.3.5]
at org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32) ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
at sink.ESSink.process(ESSink.java:77) ~[classes/:?]
at sink.ESSink.process(ESSink.java:25) ~[classes/:?]
at org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232) ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) [flink-runtime_2.10-1.1.1.jar:1.1.1]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED


We try to add mapping request to elastic search. We cannot access to client attribute (it is private) in elasticsearch class.


Is there any way to overcome this problem.


Thanks,


Ozan



________________________________
Gönderen: Till Rohrmann <[hidden email]>
Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34
Kime: [hidden email]
Bilgi: Aljoscha Krettek
Konu: Re: No support for request PutMappingRequest

Hi Ozan,

I'm not super experienced with Flink's elasticsearch connector, but could
you post the complete stack trace to figure out where the problem comes
from?

I've also pulled in Aljoscha, the original author of the elasticsearch
sink. Maybe he can give you a detailed answer.

Cheers,
Till

On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote:

> Hi everyone,
>
>
> We are trying to use elasticsearch (2.x) connector for Flink application.
> However, we encounter a problem when we try to add mapping to elasticsearch
> index.
>
>
> The error message is below when we run the Flink application.
>
>
>  No support for request [org.elasticsearch.action.
> admin.indices.mapping.put.PutMappingRequest]
>
>
> We are using "putMappingRequest" function for mapping. In Flink, is there
> any way to add mapping to indexes in elasticsearch?
>
>
> If not, can we contribute for adding mapping feature to elasticsearch
> connector?
>
Reply | Threaded
Open this post in threaded view
|

Re: No support for request PutMappingRequest

Aljoscha Krettek-2
Hi,
I think PutMappingRequest is a request that can only be sent using
IndicesAdminClient. In my understanding this is an administrative command
that isn't related to actually storing data in an index.

What are you trying to store with the PutMappingRequest?

Cheers,
Aljoscha

On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <[hidden email]> wrote:

> Hi,
>
>
> I am sending the error message below;
>
>
> java.lang.IllegalArgumentException: No support for request
> [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b
> ]
> at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at
> org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at
> org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32)
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> at sink.ESSink.process(ESSink.java:77) ~[classes/:?]
> at sink.ESSink.process(ESSink.java:25) ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232)
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> [flink-runtime_2.10-1.1.1.jar:1.1.1]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED
>
>
> We try to add mapping request to elastic search. We cannot access to
> client attribute (it is private) in elasticsearch class.
>
>
> Is there any way to overcome this problem.
>
>
> Thanks,
>
>
> Ozan
>
>
>
> ________________________________
> Gönderen: Till Rohrmann <[hidden email]>
> Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34
> Kime: [hidden email]
> Bilgi: Aljoscha Krettek
> Konu: Re: No support for request PutMappingRequest
>
> Hi Ozan,
>
> I'm not super experienced with Flink's elasticsearch connector, but could
> you post the complete stack trace to figure out where the problem comes
> from?
>
> I've also pulled in Aljoscha, the original author of the elasticsearch
> sink. Maybe he can give you a detailed answer.
>
> Cheers,
> Till
>
> On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote:
>
> > Hi everyone,
> >
> >
> > We are trying to use elasticsearch (2.x) connector for Flink application.
> > However, we encounter a problem when we try to add mapping to
> elasticsearch
> > index.
> >
> >
> > The error message is below when we run the Flink application.
> >
> >
> >  No support for request [org.elasticsearch.action.
> > admin.indices.mapping.put.PutMappingRequest]
> >
> >
> > We are using "putMappingRequest" function for mapping. In Flink, is there
> > any way to add mapping to indexes in elasticsearch?
> >
> >
> > If not, can we contribute for adding mapping feature to elasticsearch
> > connector?
> >
>
Reply | Threaded
Open this post in threaded view
|

Ynt: No support for request PutMappingRequest

Ozan DENİZ
Hi Aljoscha,


We are trying to add sortable feature for elasticsearch. To do this, we need to add mapping to index.


We try to sort some fields in elasticsearch. To make it our json format should like this;


"tweet": {
"type": "string",
"analyzer": "english",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}



We should add "not_analyzed" to map.


    private XContentBuilder buildMapping( String typeName )
    {

        XContentBuilder mapping = null;
        try
        {
            mapping = jsonBuilder()
                    .startObject()
                    .startObject(typeName)
                    .startObject("properties")


                    .startObject(LogFields.MESSAGE)
                    .field("type","string")
                    .field("analyzer", "standard")
                    .startObject("fields")
                    .startObject("raw")
                    .field("type","string")
                    .field("index", "not_analyzed")
                    .endObject()
                    .endObject()
                    .endObject()

                    .endObject()
                    .endObject()
                    .endObject();
        }
        catch ( IOException e )
        {
            e.printStackTrace();
        }

        return mapping;
    }




________________________________
Gönderen: Aljoscha Krettek <[hidden email]>
Gönderildi: 26 Eylül 2016 Pazartesi 16:27:12
Kime: [hidden email]
Konu: Re: No support for request PutMappingRequest

Hi,
I think PutMappingRequest is a request that can only be sent using
IndicesAdminClient. In my understanding this is an administrative command
that isn't related to actually storing data in an index.

What are you trying to store with the PutMappingRequest?

Cheers,
Aljoscha

On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <[hidden email]> wrote:

> Hi,
>
>
> I am sending the error message below;
>
>
> java.lang.IllegalArgumentException: No support for request
> [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b
> ]
> at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at
> org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
> ~[elasticsearch-2.3.5.jar:2.3.5]
> at
> org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32)
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> at sink.ESSink.process(ESSink.java:77) ~[classes/:?]
> at sink.ESSink.process(ESSink.java:25) ~[classes/:?]
> at
> org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232)
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> at
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266)
> ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> [flink-runtime_2.10-1.1.1.jar:1.1.1]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED
>
>
> We try to add mapping request to elastic search. We cannot access to
> client attribute (it is private) in elasticsearch class.
>
>
> Is there any way to overcome this problem.
>
>
> Thanks,
>
>
> Ozan
>
>
>
> ________________________________
> Gönderen: Till Rohrmann <[hidden email]>
> Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34
> Kime: [hidden email]
> Bilgi: Aljoscha Krettek
> Konu: Re: No support for request PutMappingRequest
>
> Hi Ozan,
>
> I'm not super experienced with Flink's elasticsearch connector, but could
> you post the complete stack trace to figure out where the problem comes
> from?
>
> I've also pulled in Aljoscha, the original author of the elasticsearch
> sink. Maybe he can give you a detailed answer.
>
> Cheers,
> Till
>
> On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote:
>
> > Hi everyone,
> >
> >
> > We are trying to use elasticsearch (2.x) connector for Flink application.
> > However, we encounter a problem when we try to add mapping to
> elasticsearch
> > index.
> >
> >
> > The error message is below when we run the Flink application.
> >
> >
> >  No support for request [org.elasticsearch.action.
> > admin.indices.mapping.put.PutMappingRequest]
> >
> >
> > We are using "putMappingRequest" function for mapping. In Flink, is there
> > any way to add mapping to indexes in elasticsearch?
> >
> >
> > If not, can we contribute for adding mapping feature to elasticsearch
> > connector?
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: No support for request PutMappingRequest

Aljoscha Krettek-2
Hi,
the mapping should not be updated in the Flink sink. According to the
documentation the mapping is a setting on an index that should not be
changed after an index was created and some documents were added to that
index:
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

I think you have to set your mappings/settings on the index before you
start your Flink job. The Flink sink is only meant for adding elements to
an index.

Cheers,
Aljoscha

On Mon, 26 Sep 2016 at 15:39 Ozan DENİZ <[hidden email]> wrote:

> Hi Aljoscha,
>
>
> We are trying to add sortable feature for elasticsearch. To do this, we
> need to add mapping to index.
>
>
> We try to sort some fields in elasticsearch. To make it our json format
> should like this;
>
>
> "tweet": {
> "type": "string",
> "analyzer": "english",
> "fields": {
> "raw": {
> "type": "string",
> "index": "not_analyzed"
> }
> }
> }
>
>
>
> We should add "not_analyzed" to map.
>
>
>     private XContentBuilder buildMapping( String typeName )
>     {
>
>         XContentBuilder mapping = null;
>         try
>         {
>             mapping = jsonBuilder()
>                     .startObject()
>                     .startObject(typeName)
>                     .startObject("properties")
>
>
>                     .startObject(LogFields.MESSAGE)
>                     .field("type","string")
>                     .field("analyzer", "standard")
>                     .startObject("fields")
>                     .startObject("raw")
>                     .field("type","string")
>                     .field("index", "not_analyzed")
>                     .endObject()
>                     .endObject()
>                     .endObject()
>
>                     .endObject()
>                     .endObject()
>                     .endObject();
>         }
>         catch ( IOException e )
>         {
>             e.printStackTrace();
>         }
>
>         return mapping;
>     }
>
>
>
>
> ________________________________
> Gönderen: Aljoscha Krettek <[hidden email]>
> Gönderildi: 26 Eylül 2016 Pazartesi 16:27:12
> Kime: [hidden email]
> Konu: Re: No support for request PutMappingRequest
>
> Hi,
> I think PutMappingRequest is a request that can only be sent using
> IndicesAdminClient. In my understanding this is an administrative command
> that isn't related to actually storing data in an index.
>
> What are you trying to store with the PutMappingRequest?
>
> Cheers,
> Aljoscha
>
> On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <[hidden email]> wrote:
>
> > Hi,
> >
> >
> > I am sending the error message below;
> >
> >
> > java.lang.IllegalArgumentException: No support for request
> >
> [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b
> > ]
> > at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107)
> > ~[elasticsearch-2.3.5.jar:2.3.5]
> > at
> >
> org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284)
> > ~[elasticsearch-2.3.5.jar:2.3.5]
> > at
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268)
> > ~[elasticsearch-2.3.5.jar:2.3.5]
> > at
> org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
> > ~[elasticsearch-2.3.5.jar:2.3.5]
> > at
> >
> org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32)
> >
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> > at sink.ESSink.process(ESSink.java:77) ~[classes/:?]
> > at sink.ESSink.process(ESSink.java:25) ~[classes/:?]
> > at
> >
> org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232)
> >
> ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT]
> > at
> >
> org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39)
> > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> > at org.apache.flink.streaming.runtime.io
> .StreamInputProcessor.processInput(StreamInputProcessor.java:176)
> > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> > at
> >
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> > at
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266)
> > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1]
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> > [flink-runtime_2.10-1.1.1.jar:1.1.1]
> > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
> > 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED
> >
> >
> > We try to add mapping request to elastic search. We cannot access to
> > client attribute (it is private) in elasticsearch class.
> >
> >
> > Is there any way to overcome this problem.
> >
> >
> > Thanks,
> >
> >
> > Ozan
> >
> >
> >
> > ________________________________
> > Gönderen: Till Rohrmann <[hidden email]>
> > Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34
> > Kime: [hidden email]
> > Bilgi: Aljoscha Krettek
> > Konu: Re: No support for request PutMappingRequest
> >
> > Hi Ozan,
> >
> > I'm not super experienced with Flink's elasticsearch connector, but could
> > you post the complete stack trace to figure out where the problem comes
> > from?
> >
> > I've also pulled in Aljoscha, the original author of the elasticsearch
> > sink. Maybe he can give you a detailed answer.
> >
> > Cheers,
> > Till
> >
> > On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]>
> wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > We are trying to use elasticsearch (2.x) connector for Flink
> application.
> > > However, we encounter a problem when we try to add mapping to
> > elasticsearch
> > > index.
> > >
> > >
> > > The error message is below when we run the Flink application.
> > >
> > >
> > >  No support for request [org.elasticsearch.action.
> > > admin.indices.mapping.put.PutMappingRequest]
> > >
> > >
> > > We are using "putMappingRequest" function for mapping. In Flink, is
> there
> > > any way to add mapping to indexes in elasticsearch?
> > >
> > >
> > > If not, can we contribute for adding mapping feature to elasticsearch
> > > connector?
> > >
> >
>