I have a question regarding the future direction of the queryable state feature.
We are current using this feature in production implemented in a RichFlatMap. It is doing what we need it to do at the scale we need it done in, with the resources we have assigned to it. Win. However, we'd also like to use this feature in conjunction with Flink's windowing. The "Rich" execution environment is not exposed in any of the windows / triggers / apply hierarchy, so we cannot expose any of the state managed within the windows outside of Flink. Many of our use cases require us to have access to values as they are being accumulated, as well as the aggregated result. We can get by with the RichFlatMap for now. I'd like some clarification as to whether or not the queryable state feature is going to be extended to the windowing components for the next milestone release. This will determine our Flink development milestones for the next few months. From consulting the open items in JIRA, it does not look like it is on the docket. I'd be more than willing to help out implementing this feature, but I don't think I have the experience to submit this change on my own. |
I think adding this could be as simple as adding a makeWindowContentsQueryable(String) call to WindowedStream. In there, we create the StateDescriptor for keeping the window contents so we can call setQueryable() on the StateDescriptor and set the name under which it should be accessible. If you want you can create an issue and change this. I can help you if you have any questions.
For custom Triggers, I think you can also call setQueryable() on a StateDescriptor. The tricky thing when it comes to windows is that state is scoped to a window, so the querying logic has to take that into account. I also cc’ed Ufuk and Nico, who probably know best what’s going on with queryable state. > On 16 Mar 2017, at 15:33, Joe Olson <[hidden email]> wrote: > > I have a question regarding the future direction of the queryable state feature. > > > We are current using this feature in production implemented in a RichFlatMap. It is doing what we need it to do at the scale we need it done in, with the resources we have assigned to it. Win. > > > However, we'd also like to use this feature in conjunction with Flink's windowing. The "Rich" execution environment is not exposed in any of the windows / triggers / apply hierarchy, so we cannot expose any of the state managed within the windows outside of Flink. Many of our use cases require us to have access to values as they are being accumulated, as well as the aggregated result. > > > We can get by with the RichFlatMap for now. I'd like some clarification as to whether or not the queryable state feature is going to be extended to the windowing components for the next milestone release. This will determine our Flink development milestones for the next few months. From consulting the open items in JIRA, it does not look like it is on the docket. > > > I'd be more than willing to help out implementing this feature, but I don't think I have the experience to submit this change on my own. |
On Mon, Mar 20, 2017 at 3:25 PM, Aljoscha Krettek <[hidden email]> wrote:
> I think adding this could be as simple as adding a makeWindowContentsQueryable(String) call to WindowedStream. In there, we create the StateDescriptor for keeping the window contents so we can call setQueryable() on the StateDescriptor and set the name under which it should be accessible. If you want you can create an issue and change this. I can help you if you have any questions. +1 I would only allow this for incrementally aggregating windows in the first version (reduce, fold, etc.). Does this make sense for your use case Joe? Otherwise, the state might grow quite large and for session windows identifying the session is quite tricky. We only need to watch out for the fast path implementation (using AbstractAlignedProcessingTimeWindowOperator) that cannot be queried. If you would like to work on this, I would also be happy to work with you and create the initial JIRA writing down things that we should watch out for. – Ufuk |
Free forum by Nabble | Edit this page |