[jira] [Created] (FLINK-2727) Add a base class for MessageQueue-with-acknowledgement sources

[Shang Yuanchun (Jira)]

Stephan Ewen created FLINK-2727:
-----------------------------------

             Summary: Add a base class for MessageQueue-with-acknowledgement sources
                 Key: FLINK-2727
                 URL: https://issues.apache.org/jira/browse/FLINK-2727
             Project: Flink
          Issue Type: New Feature
          Components: Streaming
    Affects Versions: 0.10
            Reporter: Stephan Ewen
            Assignee: Stephan Ewen


Several message queues (RabbitMQ, Amazon SQS) have the pattern that you retrieve messages and acknowledge them back by ID.

We can create a simple base non-parallel source that provides tooling for:
  - Collecting the IDs of elements emitted between two checkpoints
  - Persisting them with the checkpoint, respecting proper serialization
  - Acknowledging them when a checkpoint is notified of completion.

This assumes that the Message Queues retain unacknowledged messages and re-emit them after the acknowledgement period expired.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)