Elias Levy created FLINK-6472:
---------------------------------
Summary: BoundedOutOfOrdernessTimestampExtractor does not bound out of orderliness
Key: FLINK-6472
URL:
https://issues.apache.org/jira/browse/FLINK-6472 Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 1.3.0
Reporter: Elias Levy
{{BoundedOutOfOrdernessTimestampExtractor}} attempts to emit watermarks that lag behind the largest observed timestamp by a configurable time delta. It fails to so in some circumstances.
The class extends {{AssignerWithPeriodicWatermarks}}, which generates watermarks in periodic intervals. The timer for this intervals is a processing time timer.
In circumstances where there is a rush of events (restarting Flink, unpausing an upstream producer, loading events from a file, etc), many events with timestamps much larger that what the configured bound would normally allow will be sent downstream without a watermark. This can have negative effects downstream, as operators may be buffering the events waiting for a watermark to process them, thus leading the memory growth and possible out-of-memory conditions.
It is probably best to have a bounded out of orderliness extractor that is based on the punctuated timestamp extractor, so we can ensure that watermarks are generated in a timely fashion in event time, with the addition of process time timer to generate a watermark if there has been a lull in events, thus also bounding the delay of generating a watermark in processing time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Locked
