[jira] [Created] (FLINK-3037) Make the behavior of the Kafka consumer configurable if the offsets to restore from are not available

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (FLINK-3037) Make the behavior of the Kafka consumer configurable if the offsets to restore from are not available

Shang Yuanchun (Jira)
Robert Metzger created FLINK-3037:
-------------------------------------

             Summary: Make the behavior of the Kafka consumer configurable if the offsets to restore from are not available
                 Key: FLINK-3037
                 URL: https://issues.apache.org/jira/browse/FLINK-3037
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
            Reporter: Robert Metzger


Currently, if the {{FlinkKafkaConsumer}} is restoring a checkpoint and the offset is not available anymore in Kafka, its restoring according to {{auto.offset.reset}}.
This leads to inconsistent behavior (not exactly-once anymore) because the operators will not receive data in sync with the checkpoint.

With this pull request, I would like to make the behavior controllable, using a flag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)