Watermark options for TwoInputStreamOperator

[Jocean shi]

Hi all:
    Currently, The "TwoInputStreamOperator" such as
"CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
(Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
support compute watermark by two stream.
but we just need one stream to compute watermark in some case.
     For example: one stream is only rule data or control command, it
don't hava event time.
I think user has right to choose which stream to compute watermark.

Best
Ocean

[Jark Wu-2]

Hi Ocean,

You can implement your custom operator by the "TwoInputStreamOperator"
interface.
The TwoInputStreamOperator interface provides "processWatermark1" and
"processWatermark2" which handles
watermarks for left stream and right stream. You can then ignore the
watermarks from right stream and forward
watermark from left stream.

Best,
Jark

On Mon, 23 Dec 2019 at 21:47, Jocean shi <[hidden email]> wrote:

> Hi all:
>     Currently, The "TwoInputStreamOperator" such as
> "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> support compute watermark by two stream.
> but we just need one stream to compute watermark in some case.
>      For example: one stream is only rule data or control command, it
> don't hava event time.
> I think user has right to choose which stream to compute watermark.
>
> Best
> Ocean
>

[Jocean shi]

Hi Jark,

I got you. We have discussed this question in Flink Forward 2019.
I know that i can custom operator to resolve this problem.
but also has some other problems:

First,
This is a very common scene that we often meet
I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
bucause it's "protected",
It's unfriendly for new user.

Second.
One stream even don't call "assignTimestampsAndWatermarks" method,
It's mean this stream don't have time and watermark,
It don't have time and watermark why It have a hand in computing of
watermak and time default.

Best
Jark

Jark Wu <[hidden email]> 于2019年12月24日周二 下午12:08写道：

>
> Hi Ocean,
>
> You can implement your custom operator by the "TwoInputStreamOperator"
> interface.
> The TwoInputStreamOperator interface provides "processWatermark1" and
> "processWatermark2" which handles
> watermarks for left stream and right stream. You can then ignore the
> watermarks from right stream and forward
> watermark from left stream.
>
> Best,
> Jark
>
> On Mon, 23 Dec 2019 at 21:47, Jocean shi <[hidden email]> wrote:
>
> > Hi all:
> >     Currently, The "TwoInputStreamOperator" such as
> > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> > support compute watermark by two stream.
> > but we just need one stream to compute watermark in some case.
> >      For example: one stream is only rule data or control command, it
> > don't hava event time.
> > I think user has right to choose which stream to compute watermark.
> >
> > Best
> > Ocean
> >

[Congxian Qiu]

Hi

Maybe the doc[1] about  Idling Source can help there.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#idling-sources

Best,
Congxian


Jocean shi <[hidden email]> 于2019年12月24日周二 下午3:05写道：

> Hi Jark,
>
> I got you. We have discussed this question in Flink Forward 2019.
> I know that i can custom operator to resolve this problem.
> but also has some other problems:
>
> First,
> This is a very common scene that we often meet
> I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
> and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
> bucause it's "protected",
> It's unfriendly for new user.
>
> Second.
> One stream even don't call "assignTimestampsAndWatermarks" method,
> It's mean this stream don't have time and watermark,
> It don't have time and watermark why It have a hand in computing of
> watermak and time default.
>
> Best
> Jark
>
> Jark Wu <[hidden email]> 于2019年12月24日周二 下午12:08写道：
> >
> > Hi Ocean,
> >
> > You can implement your custom operator by the "TwoInputStreamOperator"
> > interface.
> > The TwoInputStreamOperator interface provides "processWatermark1" and
> > "processWatermark2" which handles
> > watermarks for left stream and right stream. You can then ignore the
> > watermarks from right stream and forward
> > watermark from left stream.
> >
> > Best,
> > Jark
> >
> > On Mon, 23 Dec 2019 at 21:47, Jocean shi <[hidden email]> wrote:
> >
> > > Hi all:
> > >     Currently, The "TwoInputStreamOperator" such as
> > > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> > > support compute watermark by two stream.
> > > but we just need one stream to compute watermark in some case.
> > >      For example: one stream is only rule data or control command, it
> > > don't hava event time.
> > > I think user has right to choose which stream to compute watermark.
> > >
> > > Best
> > > Ocean
> > >
>

[Jocean shi]

Hi Congxian,

It is not the same with Idling Source

Best
Ocean

Congxian Qiu <[hidden email]> 于2019年12月24日周二 下午4:39写道：

>
> Hi
>
> Maybe the doc[1] about  Idling Source can help there.
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#idling-sources
>
> Best,
> Congxian
>
>
> Jocean shi <[hidden email]> 于2019年12月24日周二 下午3:05写道：
>
> > Hi Jark,
> >
> > I got you. We have discussed this question in Flink Forward 2019.
> > I know that i can custom operator to resolve this problem.
> > but also has some other problems:
> >
> > First,
> > This is a very common scene that we often meet
> > I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
> > and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
> > bucause it's "protected",
> > It's unfriendly for new user.
> >
> > Second.
> > One stream even don't call "assignTimestampsAndWatermarks" method,
> > It's mean this stream don't have time and watermark,
> > It don't have time and watermark why It have a hand in computing of
> > watermak and time default.
> >
> > Best
> > Jark
> >
> > Jark Wu <[hidden email]> 于2019年12月24日周二 下午12:08写道：
> > >
> > > Hi Ocean,
> > >
> > > You can implement your custom operator by the "TwoInputStreamOperator"
> > > interface.
> > > The TwoInputStreamOperator interface provides "processWatermark1" and
> > > "processWatermark2" which handles
> > > watermarks for left stream and right stream. You can then ignore the
> > > watermarks from right stream and forward
> > > watermark from left stream.
> > >
> > > Best,
> > > Jark
> > >
> > > On Mon, 23 Dec 2019 at 21:47, Jocean shi <[hidden email]> wrote:
> > >
> > > > Hi all:
> > > >     Currently, The "TwoInputStreamOperator" such as
> > > > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > > > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only
> > > > support compute watermark by two stream.
> > > > but we just need one stream to compute watermark in some case.
> > > >      For example: one stream is only rule data or control command, it
> > > > don't hava event time.
> > > > I think user has right to choose which stream to compute watermark.
> > > >
> > > > Best
> > > > Ocean
> > > >
> >

[Victor Wong]

Hi Jocean,

How about assigning the control data stream with Long.MAX_VALUE as
watermark?

Best,
Victor

Jocean shi <[hidden email]> 于2019年12月25日周三 上午9:53写道：

> Hi Congxian,
>
> It is not the same with Idling Source
>
> Best
> Ocean
>
> Congxian Qiu <[hidden email]> 于2019年12月24日周二 下午4:39写道：
> >
> > Hi
> >
> > Maybe the doc[1] about  Idling Source can help there.
> > [1]
> >
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#idling-sources
> >
> > Best,
> > Congxian
> >
> >
> > Jocean shi <[hidden email]> 于2019年12月24日周二 下午3:05写道：
> >
> > > Hi Jark,
> > >
> > > I got you. We have discussed this question in Flink Forward 2019.
> > > I know that i can custom operator to resolve this problem.
> > > but also has some other problems:
> > >
> > > First,
> > > This is a very common scene that we often meet
> > > I have to rewrite "BroadcastConnectedStream","ConnectedStreams" ...
> > > and "TwoInputStreamOperator" and rewrite "SingleOutputStreamOperator"
> > > bucause it's "protected",
> > > It's unfriendly for new user.
> > >
> > > Second.
> > > One stream even don't call "assignTimestampsAndWatermarks" method,
> > > It's mean this stream don't have time and watermark,
> > > It don't have time and watermark why It have a hand in computing of
> > > watermak and time default.
> > >
> > > Best
> > > Jark
> > >
> > > Jark Wu <[hidden email]> 于2019年12月24日周二 下午12:08写道：
> > > >
> > > > Hi Ocean,
> > > >
> > > > You can implement your custom operator by the
> "TwoInputStreamOperator"
> > > > interface.
> > > > The TwoInputStreamOperator interface provides "processWatermark1" and
> > > > "processWatermark2" which handles
> > > > watermarks for left stream and right stream. You can then ignore the
> > > > watermarks from right stream and forward
> > > > watermark from left stream.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Mon, 23 Dec 2019 at 21:47, Jocean shi <[hidden email]>
> wrote:
> > > >
> > > > > Hi all:
> > > > >     Currently, The "TwoInputStreamOperator" such as
> > > > > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the
> > > > > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream"
> only
> > > > > support compute watermark by two stream.
> > > > > but we just need one stream to compute watermark in some case.
> > > > >      For example: one stream is only rule data or control command,
> it
> > > > > don't hava event time.
> > > > > I think user has right to choose which stream to compute watermark.
> > > > >
> > > > > Best
> > > > > Ocean
> > > > >
> > >
>

--

Best,
Victor