The right approach to store intermediate value

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

The right approach to store intermediate value

Shengnan YU
Dear Flink committers:
I am starting to learn implementing some data streaming algorithm based on Flink API. Now I am working on MisraGries algorithm. It keeps k counters to record the data summary by increment or decrement. I want to know what is the best approach to store those counters(a hashmap data structure maybe) when using DataStream API to implement the algorithm. Now I just declared a map variable in the operator, is it right or I need to use some other features like state. This may be a very basic question forgive me since I am just a beginnerT^T.

Best regards
------------------------------------------------------
Shengnan YU(於圣楠)

[hidden email]<mailto:[hidden email]>
+852 69963058

Reply | Threaded
Open this post in threaded view
|

Re: The right approach to store intermediate value

Aljoscha Krettek-2
Hi,

You should use the State API [1] for this if you want your program to be fault-tolerant. Flink will checkpoint state and make sure that it is restored in case of failures.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/state/state.html>

Best,
Aljoscha

> On 14. Nov 2017, at 05:53, Shengnan YU <[hidden email]> wrote:
>
> Dear Flink committers:
> I am starting to learn implementing some data streaming algorithm based on Flink API. Now I am working on MisraGries algorithm. It keeps k counters to record the data summary by increment or decrement. I want to know what is the best approach to store those counters(a hashmap data structure maybe) when using DataStream API to implement the algorithm. Now I just declared a map variable in the operator, is it right or I need to use some other features like state. This may be a very basic question forgive me since I am just a beginnerT^T.
>
> Best regards
> ------------------------------------------------------
> Shengnan YU(於圣楠)
>
> [hidden email]<mailto:[hidden email]>
> +852 69963058
>