Why aren't Range Scans Supported in MapState?

[Aaron Langford]

Hey Flink Community,

RocksDB natively supports range scans for keys. Is there a good story why
we don't have a way to ask for keys by range or prefix currently in
MapState?

Aaron

[Yun Tang]

Hi Aaron

We would usually compare MapState with Map in java language. And your request is not what typical Map interface but SortedMap defines.

I think it's reasonable If we introduce SortedMapState interface in Flink. However, there existed several challenges for RocksDB.

  *   RocksDB only support byte-wise comparator by default, which means those non-fixed length serializer (e.g. StringSerializer) would generate bytes with different lexicon order compared with their original strings in java. Blink had ever tried to implement this feature named OrderedBytes [1], which still has many limitations and only applied in table API.

  *   RocksDB actually support java comparator and we have also implemented it privately on RocksDB-4.2. However, the performance behaves badly.
  *   RocksDB later support native comparatot for java users [2] and we are still looking at this future although this would be a bit complex for Flink users if they want their classes could be comparable as they want.

Hope this could help you.

[1] https://github.com/apache/flink/blob/blink/flink-libraries/flink-table/src/main/java/org/apache/flink/table/typeutils/ordered/OrderedBytes.java
  *   <https://github.com/apache/flink/blob/blink/flink-libraries/flink-table/src/main/java/org/apache/flink/table/typeutils/ordered/OrderedBytes.java> [2] https://github.com/facebook/rocksdb/pull/3334

Best
Yun Tang


  *

________________________________
From: Aaron Langford <[hidden email]>
Sent: Tuesday, January 7, 2020 1:36
To: [hidden email] <[hidden email]>
Subject: Why aren't Range Scans Supported in MapState?

Hey Flink Community,

RocksDB natively supports range scans for keys. Is there a good story why
we don't have a way to ask for keys by range or prefix currently in
MapState?

Aaron