Jan Westphal created FLINK-21373:
------------------------------------
Summary: Port RabbitMQ Sink to FLIP-143 API
Key: FLINK-21373
URL:
https://issues.apache.org/jira/browse/FLINK-21373 Project: Flink
Issue Type: Improvement
Components: Connectors/ RabbitMQ
Reporter: Jan Westphal
Fix For: 1.12.0
*Structure*
The unified Sink API provides a Writer, a Committer and a GlobalCommitter. Right now we don’t see the need to use the Committer and GlobalCommitter as the Writer is sufficient to hold up to the consistencies. Since we are in the need of asynchronous RabbitMQ callbacks to know whether or not a message was published successfully and have to store unacknowledged messages in the checkpoint, there would be a large bidirectional communication and state exchange overhead between the Writer and the Committer.
*At-most-once*
The Writer receives a message from Flink and simply publishes it to RabbitMQ. The current RabbitMQ Sink only provides this mode.
*At-least-once*
The objective here is, to receive an acknowledgement by RabbitMQ for published messages. Therefore, before publishing a message, we store the message in a Map with the sequence number as its key. If the message is acknowledged by RabbitMQ we can remove it from the Map. If we don’t receive an acknowledgement for a certain amount of time (or a RabbitMQ specific so called negative acknowledgement) we will try to resend the message when doing a checkpoint.
*Exactly-once*
On checkpointing we send all messages by Flink in transaction mode to RabbitMQ. This way, all the messages get sent or are rolled back on failure. All messages that are not sent successfully are written to the checkpoint and are tried to be sent with the next checkpoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
