[DISCUSS] Watermark propagation with Sink API

Posted by Eron Wright-3 on
URL: http://deprecated-apache-flink-mailing-list-archive.368.s1.nabble.com/DISCUSS-Watermark-propagation-with-Sink-API-tp50784.html

I would like to propose an enhancement to the Sink API, the ability to
receive upstream watermarks.   I'm aware that the sink context provides the
current watermark for a given record.  I'd like to be able to write a sink
function that is invoked whenever the watermark changes.  Out of scope
would be event-time timers (since sinks aren't keyed).

For context, imagine that a stream storage system had the ability to
persist watermarks in addition to ordinary elements, e.g. to serve as
source watermarks in a downstream processor.  Ideally one could compose a
multi-stage, event-driven application, with watermarks flowing end-to-end
without need for a heuristics-based watermark at each stage.

The specific proposal would be a new method on `SinkFunction` and/or on
`SinkWriter`, called 'processWatermark' or 'writeWatermark', with a default
implementation that does nothing.

Thoughts?

Thanks!
Eron Wright
StreamNative