Reading RocksDB contents from outside of Flink

[Shilpa Deshpande]

Hello All,

I am new to Apache Flink. In my company we are thinking of using Flink to
perform transformation of the data. The source of the data is Apache Kafka
topics. Each message that we receive on Kafka topic, we want to transform
it and store it on RocksDB. The messages can come out of order. We would
like to store the transformed output in RocksDB using Flink and retrieve it
from a Spring Boot application. The reason for Spring Boot Application is
outbound messaging involves some stitching as well as ordering of the data.
Is it possible and advisable to read data from RocksDB from Spring Boot
Application? Spring Boot Application will not change the data. The reason
we think Flink can help us is because we might be receiving millions of
messages during the day so want to make sure use a technology that scales
well. If you have a snippet of code to achieve this, please share it with
us.

Thank you for your inputs!

Regards,
Shilpa

[taher koitawala-2]

Hi Shilpa,
       The easiest way to do this is the make the Rocks DB state queryable.
Then use the Flink queryable state client to access the state you have
created.


Regards
Taher Koitawala

On Tue, Jul 30, 2019, 4:58 PM Shilpa Deshpande <[hidden email]> wrote:

> Hello All,
>
> I am new to Apache Flink. In my company we are thinking of using Flink to
> perform transformation of the data. The source of the data is Apache Kafka
> topics. Each message that we receive on Kafka topic, we want to transform
> it and store it on RocksDB. The messages can come out of order. We would
> like to store the transformed output in RocksDB using Flink and retrieve it
> from a Spring Boot application. The reason for Spring Boot Application is
> outbound messaging involves some stitching as well as ordering of the data.
> Is it possible and advisable to read data from RocksDB from Spring Boot
> Application? Spring Boot Application will not change the data. The reason
> we think Flink can help us is because we might be receiving millions of
> messages during the day so want to make sure use a technology that scales
> well. If you have a snippet of code to achieve this, please share it with
> us.
>
> Thank you for your inputs!
>
> Regards,
> Shilpa
>

[Stephan Ewen]

Hi!

Are you looking for online access or offline access?

For online access, you can to key lookups via queryable state.

For offline access, you can read and write rocksDB state using the new
state processor API in Flink 1.9
https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/state_processor_api.html


Best,
Stephan


On Tue, Jul 30, 2019 at 2:39 PM taher koitawala <[hidden email]> wrote:

> Hi Shilpa,
>        The easiest way to do this is the make the Rocks DB state queryable.
> Then use the Flink queryable state client to access the state you have
> created.
>
>
> Regards
> Taher Koitawala
>
> On Tue, Jul 30, 2019, 4:58 PM Shilpa Deshpande <[hidden email]>
> wrote:
>
> > Hello All,
> >
> > I am new to Apache Flink. In my company we are thinking of using Flink to
> > perform transformation of the data. The source of the data is Apache
> Kafka
> > topics. Each message that we receive on Kafka topic, we want to transform
> > it and store it on RocksDB. The messages can come out of order. We would
> > like to store the transformed output in RocksDB using Flink and retrieve
> it
> > from a Spring Boot application. The reason for Spring Boot Application is
> > outbound messaging involves some stitching as well as ordering of the
> data.
> > Is it possible and advisable to read data from RocksDB from Spring Boot
> > Application? Spring Boot Application will not change the data. The reason
> > we think Flink can help us is because we might be receiving millions of
> > messages during the day so want to make sure use a technology that scales
> > well. If you have a snippet of code to achieve this, please share it with
> > us.
> >
> > Thank you for your inputs!
> >
> > Regards,
> > Shilpa
> >
>