update data from two sources with Flink

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

update data from two sources with Flink

ziv-3
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: update data from two sources with Flink

Tzu-Li (Gordon) Tai
Hi Ziv,

From what I think what you mean, you can do this by implementing a CoFlatMapFunction. Each of the two inputs of the CoFlatMapFunction would be your field-contributing sources.
When either input first receives field values of a new object, store that uncompleted object as state of the CoFlatMapFunction.
When you receive the other remaining field values from the other input, read the object from the state, complete it and then emit it from the CoFlatMapFunction. You should also remember to clear that state if it will no longer be used.
A downstream sink can be added to write the completed object to some external system.

Hope that helps!
Gordon

On 10 September 2017 at 7:52:35 PM, ziv ([hidden email]) wrote:

Hi,

I have two sources that both contribute to build fields of objects. When
building an object is completed I sink it out.

Can anyone suggest the effective way for such a program with Flink?

Ziv




--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/