State Processor API SQL State

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

State Processor API SQL State

Dominik Wosiński
Hey,
Is it currently possible to obtain the state that was created by SQL query
via the State Processor API? I am able to load the checkpoint via the State
Processor API, but I wasn't able to think of a way to access the internal
state of my JOIN Query.

Best Regards,
Dom.
Reply | Threaded
Open this post in threaded view
|

Re: State Processor API SQL State

Yun Tang
Hi Dom,

+ user mail list

Once you got to know the state descriptor, I think you could query the join state. The state name is easy to get via [1], it should be "left-records" and "right-records", and you could check what kind of join and whether has unique key to decide what kind of state (value state or map state). The last part is to find what the rowData type is in your join, and maybe other SQL guys could answer this or you might find it by yourself by dumping the memory of your taskmanager.

[1] https://github.com/apache/flink/blob/7a7c87096ab76f416cd7c393240faa8454db36f0/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/stream/StreamingJoinOperator.java#L83

Best
Yun Tang



________________________________
From: Dominik Wosiński <[hidden email]>
Sent: Tuesday, December 1, 2020 21:05
To: dev <[hidden email]>
Subject: State Processor API SQL State

Hey,
Is it currently possible to obtain the state that was created by SQL query
via the State Processor API? I am able to load the checkpoint via the State
Processor API, but I wasn't able to think of a way to access the internal
state of my JOIN Query.

Best Regards,
Dom.
Reply | Threaded
Open this post in threaded view
|

Re: State Processor API SQL State

Dominik Wosiński
Hey,
Thanks a lot for that.
Best Regards,
Dom.

śr., 2 gru 2020 o 03:45 Yun Tang <[hidden email]> napisał(a):

> Hi Dom,
>
> + user mail list
>
> Once you got to know the state descriptor, I think you could query the
> join state. The state name is easy to get via [1], it should be
> "left-records" and "right-records", and you could check what kind of join
> and whether has unique key to decide what kind of state (value state or map
> state). The last part is to find what the rowData type is in your join, and
> maybe other SQL guys could answer this or you might find it by yourself by
> dumping the memory of your taskmanager.
>
> [1]
> https://github.com/apache/flink/blob/7a7c87096ab76f416cd7c393240faa8454db36f0/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/stream/StreamingJoinOperator.java#L83
>
> Best
> Yun Tang
>
>
>
> ________________________________
> From: Dominik Wosiński <[hidden email]>
> Sent: Tuesday, December 1, 2020 21:05
> To: dev <[hidden email]>
> Subject: State Processor API SQL State
>
> Hey,
> Is it currently possible to obtain the state that was created by SQL query
> via the State Processor API? I am able to load the checkpoint via the State
> Processor API, but I wasn't able to think of a way to access the internal
> state of my JOIN Query.
>
> Best Regards,
> Dom.
>