BroadcastState enforce processing

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

BroadcastState enforce processing

Dominik Wosiński
Hey,
I have a question since I have observed the following situation:

We have two streams  A and B that will be read from Kafka. Let's say we
have a set of rules for processing that is stored in A and B is the stream
that we will process.
Since there is no guarantee that elements from A will be processed before
elements from B, we have no actual guarantee that we will have any rules
when `processElement()` will be invoked for any element from B.

The question is, if there is any possibility apart from hardcoding some
starting rules, to enforce that we will start processing A before B ??
Let's assume that A has only one record for this case.


Best Regards,
Dom.
Reply | Threaded
Open this post in threaded view
|

Re: BroadcastState enforce processing

Yun Gao
    Hi Dominik,

        If I understand the scenario correctly, I think one possible solution is to implement one specified TwoInputStreamOperator directly, and also implements the InputSelectable interface. Then this operator could control the priority its two inputs by return proper InputSelection indication. The operator could be used with A.connect(B).transform(...) method.  

      Best,
      Yun


------------------------------------------------------------------
From:Dominik Wosiński <[hidden email]>
Send Time:2020 Jan. 20 (Mon.) 18:58
To:dev <[hidden email]>
Subject:BroadcastState enforce processing

Hey,
I have a question since I have observed the following situation:

We have two streams  A and B that will be read from Kafka. Let's say we
have a set of rules for processing that is stored in A and B is the stream
that we will process.
Since there is no guarantee that elements from A will be processed before
elements from B, we have no actual guarantee that we will have any rules
when `processElement()` will be invoked for any element from B.

The question is, if there is any possibility apart from hardcoding some
starting rules, to enforce that we will start processing A before B ??
Let's assume that A has only one record for this case.


Best Regards,
Dom.