Hi everyone,
We are trying to use elasticsearch (2.x) connector for Flink application. However, we encounter a problem when we try to add mapping to elasticsearch index. The error message is below when we run the Flink application. No support for request [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest] We are using "putMappingRequest" function for mapping. In Flink, is there any way to add mapping to indexes in elasticsearch? If not, can we contribute for adding mapping feature to elasticsearch connector? |
Hi Ozan,
I'm not super experienced with Flink's elasticsearch connector, but could you post the complete stack trace to figure out where the problem comes from? I've also pulled in Aljoscha, the original author of the elasticsearch sink. Maybe he can give you a detailed answer. Cheers, Till On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote: > Hi everyone, > > > We are trying to use elasticsearch (2.x) connector for Flink application. > However, we encounter a problem when we try to add mapping to elasticsearch > index. > > > The error message is below when we run the Flink application. > > > No support for request [org.elasticsearch.action. > admin.indices.mapping.put.PutMappingRequest] > > > We are using "putMappingRequest" function for mapping. In Flink, is there > any way to add mapping to indexes in elasticsearch? > > > If not, can we contribute for adding mapping feature to elasticsearch > connector? > |
Hi,
I am sending the error message below; java.lang.IllegalArgumentException: No support for request [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b] at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107) ~[elasticsearch-2.3.5.jar:2.3.5] at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284) ~[elasticsearch-2.3.5.jar:2.3.5] at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268) ~[elasticsearch-2.3.5.jar:2.3.5] at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264) ~[elasticsearch-2.3.5.jar:2.3.5] at org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32) ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] at sink.ESSink.process(ESSink.java:77) ~[classes/:?] at sink.ESSink.process(ESSink.java:25) ~[classes/:?] at org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232) ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266) ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) [flink-runtime_2.10-1.1.1.jar:1.1.1] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED We try to add mapping request to elastic search. We cannot access to client attribute (it is private) in elasticsearch class. Is there any way to overcome this problem. Thanks, Ozan ________________________________ Gönderen: Till Rohrmann <[hidden email]> Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34 Kime: [hidden email] Bilgi: Aljoscha Krettek Konu: Re: No support for request PutMappingRequest Hi Ozan, I'm not super experienced with Flink's elasticsearch connector, but could you post the complete stack trace to figure out where the problem comes from? I've also pulled in Aljoscha, the original author of the elasticsearch sink. Maybe he can give you a detailed answer. Cheers, Till On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote: > Hi everyone, > > > We are trying to use elasticsearch (2.x) connector for Flink application. > However, we encounter a problem when we try to add mapping to elasticsearch > index. > > > The error message is below when we run the Flink application. > > > No support for request [org.elasticsearch.action. > admin.indices.mapping.put.PutMappingRequest] > > > We are using "putMappingRequest" function for mapping. In Flink, is there > any way to add mapping to indexes in elasticsearch? > > > If not, can we contribute for adding mapping feature to elasticsearch > connector? > |
Hi,
I think PutMappingRequest is a request that can only be sent using IndicesAdminClient. In my understanding this is an administrative command that isn't related to actually storing data in an index. What are you trying to store with the PutMappingRequest? Cheers, Aljoscha On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <[hidden email]> wrote: > Hi, > > > I am sending the error message below; > > > java.lang.IllegalArgumentException: No support for request > [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b > ] > at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107) > ~[elasticsearch-2.3.5.jar:2.3.5] > at > org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284) > ~[elasticsearch-2.3.5.jar:2.3.5] > at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268) > ~[elasticsearch-2.3.5.jar:2.3.5] > at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264) > ~[elasticsearch-2.3.5.jar:2.3.5] > at > org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32) > ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] > at sink.ESSink.process(ESSink.java:77) ~[classes/:?] > at sink.ESSink.process(ESSink.java:25) ~[classes/:?] > at > org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232) > ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) > [flink-runtime_2.10-1.1.1.jar:1.1.1] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] > 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED > > > We try to add mapping request to elastic search. We cannot access to > client attribute (it is private) in elasticsearch class. > > > Is there any way to overcome this problem. > > > Thanks, > > > Ozan > > > > ________________________________ > Gönderen: Till Rohrmann <[hidden email]> > Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34 > Kime: [hidden email] > Bilgi: Aljoscha Krettek > Konu: Re: No support for request PutMappingRequest > > Hi Ozan, > > I'm not super experienced with Flink's elasticsearch connector, but could > you post the complete stack trace to figure out where the problem comes > from? > > I've also pulled in Aljoscha, the original author of the elasticsearch > sink. Maybe he can give you a detailed answer. > > Cheers, > Till > > On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote: > > > Hi everyone, > > > > > > We are trying to use elasticsearch (2.x) connector for Flink application. > > However, we encounter a problem when we try to add mapping to > elasticsearch > > index. > > > > > > The error message is below when we run the Flink application. > > > > > > No support for request [org.elasticsearch.action. > > admin.indices.mapping.put.PutMappingRequest] > > > > > > We are using "putMappingRequest" function for mapping. In Flink, is there > > any way to add mapping to indexes in elasticsearch? > > > > > > If not, can we contribute for adding mapping feature to elasticsearch > > connector? > > > |
Hi Aljoscha,
We are trying to add sortable feature for elasticsearch. To do this, we need to add mapping to index. We try to sort some fields in elasticsearch. To make it our json format should like this; "tweet": { "type": "string", "analyzer": "english", "fields": { "raw": { "type": "string", "index": "not_analyzed" } } } We should add "not_analyzed" to map. private XContentBuilder buildMapping( String typeName ) { XContentBuilder mapping = null; try { mapping = jsonBuilder() .startObject() .startObject(typeName) .startObject("properties") .startObject(LogFields.MESSAGE) .field("type","string") .field("analyzer", "standard") .startObject("fields") .startObject("raw") .field("type","string") .field("index", "not_analyzed") .endObject() .endObject() .endObject() .endObject() .endObject() .endObject(); } catch ( IOException e ) { e.printStackTrace(); } return mapping; } ________________________________ Gönderen: Aljoscha Krettek <[hidden email]> Gönderildi: 26 Eylül 2016 Pazartesi 16:27:12 Kime: [hidden email] Konu: Re: No support for request PutMappingRequest Hi, I think PutMappingRequest is a request that can only be sent using IndicesAdminClient. In my understanding this is an administrative command that isn't related to actually storing data in an index. What are you trying to store with the PutMappingRequest? Cheers, Aljoscha On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <[hidden email]> wrote: > Hi, > > > I am sending the error message below; > > > java.lang.IllegalArgumentException: No support for request > [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b > ] > at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107) > ~[elasticsearch-2.3.5.jar:2.3.5] > at > org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284) > ~[elasticsearch-2.3.5.jar:2.3.5] > at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268) > ~[elasticsearch-2.3.5.jar:2.3.5] > at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264) > ~[elasticsearch-2.3.5.jar:2.3.5] > at > org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32) > ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] > at sink.ESSink.process(ESSink.java:77) ~[classes/:?] > at sink.ESSink.process(ESSink.java:25) ~[classes/:?] > at > org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232) > ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] > at > org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:176) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266) > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) > [flink-runtime_2.10-1.1.1.jar:1.1.1] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] > 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED > > > We try to add mapping request to elastic search. We cannot access to > client attribute (it is private) in elasticsearch class. > > > Is there any way to overcome this problem. > > > Thanks, > > > Ozan > > > > ________________________________ > Gönderen: Till Rohrmann <[hidden email]> > Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34 > Kime: [hidden email] > Bilgi: Aljoscha Krettek > Konu: Re: No support for request PutMappingRequest > > Hi Ozan, > > I'm not super experienced with Flink's elasticsearch connector, but could > you post the complete stack trace to figure out where the problem comes > from? > > I've also pulled in Aljoscha, the original author of the elasticsearch > sink. Maybe he can give you a detailed answer. > > Cheers, > Till > > On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> wrote: > > > Hi everyone, > > > > > > We are trying to use elasticsearch (2.x) connector for Flink application. > > However, we encounter a problem when we try to add mapping to > elasticsearch > > index. > > > > > > The error message is below when we run the Flink application. > > > > > > No support for request [org.elasticsearch.action. > > admin.indices.mapping.put.PutMappingRequest] > > > > > > We are using "putMappingRequest" function for mapping. In Flink, is there > > any way to add mapping to indexes in elasticsearch? > > > > > > If not, can we contribute for adding mapping feature to elasticsearch > > connector? > > > |
Hi,
the mapping should not be updated in the Flink sink. According to the documentation the mapping is a setting on an index that should not be changed after an index was created and some documents were added to that index: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html I think you have to set your mappings/settings on the index before you start your Flink job. The Flink sink is only meant for adding elements to an index. Cheers, Aljoscha On Mon, 26 Sep 2016 at 15:39 Ozan DENİZ <[hidden email]> wrote: > Hi Aljoscha, > > > We are trying to add sortable feature for elasticsearch. To do this, we > need to add mapping to index. > > > We try to sort some fields in elasticsearch. To make it our json format > should like this; > > > "tweet": { > "type": "string", > "analyzer": "english", > "fields": { > "raw": { > "type": "string", > "index": "not_analyzed" > } > } > } > > > > We should add "not_analyzed" to map. > > > private XContentBuilder buildMapping( String typeName ) > { > > XContentBuilder mapping = null; > try > { > mapping = jsonBuilder() > .startObject() > .startObject(typeName) > .startObject("properties") > > > .startObject(LogFields.MESSAGE) > .field("type","string") > .field("analyzer", "standard") > .startObject("fields") > .startObject("raw") > .field("type","string") > .field("index", "not_analyzed") > .endObject() > .endObject() > .endObject() > > .endObject() > .endObject() > .endObject(); > } > catch ( IOException e ) > { > e.printStackTrace(); > } > > return mapping; > } > > > > > ________________________________ > Gönderen: Aljoscha Krettek <[hidden email]> > Gönderildi: 26 Eylül 2016 Pazartesi 16:27:12 > Kime: [hidden email] > Konu: Re: No support for request PutMappingRequest > > Hi, > I think PutMappingRequest is a request that can only be sent using > IndicesAdminClient. In my understanding this is an administrative command > that isn't related to actually storing data in an index. > > What are you trying to store with the PutMappingRequest? > > Cheers, > Aljoscha > > On Mon, 26 Sep 2016 at 15:16 Ozan DENİZ <[hidden email]> wrote: > > > Hi, > > > > > > I am sending the error message below; > > > > > > java.lang.IllegalArgumentException: No support for request > > > [org.elasticsearch.action.admin.indices.mapping.put.PutMappingRequest@3ed05d8b > > ] > > at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:107) > > ~[elasticsearch-2.3.5.jar:2.3.5] > > at > > > org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:284) > > ~[elasticsearch-2.3.5.jar:2.3.5] > > at > org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268) > > ~[elasticsearch-2.3.5.jar:2.3.5] > > at > org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264) > > ~[elasticsearch-2.3.5.jar:2.3.5] > > at > > > org.apache.flink.streaming.connectors.elasticsearch2.BulkProcessorIndexer.add(BulkProcessorIndexer.java:32) > > > ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] > > at sink.ESSink.process(ESSink.java:77) ~[classes/:?] > > at sink.ESSink.process(ESSink.java:25) ~[classes/:?] > > at > > > org.apache.flink.streaming.connectors.elasticsearch2.ElasticsearchSink.invoke(ElasticsearchSink.java:232) > > > ~[flink-connector-elasticsearch2_2.10-1.2-20160926.041955-45.jar:1.2-SNAPSHOT] > > at > > > org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:39) > > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > > at org.apache.flink.streaming.runtime.io > .StreamInputProcessor.processInput(StreamInputProcessor.java:176) > > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > > at > > > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66) > > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > > at > > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266) > > ~[flink-streaming-java_2.10-1.1.1.jar:1.1.1] > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) > > [flink-runtime_2.10-1.1.1.jar:1.1.1] > > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] > > 09/26/2016 16:14:05 Sink: Unnamed(1/1) switched to FAILED > > > > > > We try to add mapping request to elastic search. We cannot access to > > client attribute (it is private) in elasticsearch class. > > > > > > Is there any way to overcome this problem. > > > > > > Thanks, > > > > > > Ozan > > > > > > > > ________________________________ > > Gönderen: Till Rohrmann <[hidden email]> > > Gönderildi: 26 Eylül 2016 Pazartesi 13:30:34 > > Kime: [hidden email] > > Bilgi: Aljoscha Krettek > > Konu: Re: No support for request PutMappingRequest > > > > Hi Ozan, > > > > I'm not super experienced with Flink's elasticsearch connector, but could > > you post the complete stack trace to figure out where the problem comes > > from? > > > > I've also pulled in Aljoscha, the original author of the elasticsearch > > sink. Maybe he can give you a detailed answer. > > > > Cheers, > > Till > > > > On Fri, Sep 23, 2016 at 1:41 PM, Ozan DENİZ <[hidden email]> > wrote: > > > > > Hi everyone, > > > > > > > > > We are trying to use elasticsearch (2.x) connector for Flink > application. > > > However, we encounter a problem when we try to add mapping to > > elasticsearch > > > index. > > > > > > > > > The error message is below when we run the Flink application. > > > > > > > > > No support for request [org.elasticsearch.action. > > > admin.indices.mapping.put.PutMappingRequest] > > > > > > > > > We are using "putMappingRequest" function for mapping. In Flink, is > there > > > any way to add mapping to indexes in elasticsearch? > > > > > > > > > If not, can we contribute for adding mapping feature to elasticsearch > > > connector? > > > > > > |
Free forum by Nabble | Edit this page |