Tzu-Li (Gordon) Tai created FLINK-6109:
------------------------------------------
Summary: Add "consumer lag" report metric to FlinkKafkaConsumer
Key: FLINK-6109
URL:
https://issues.apache.org/jira/browse/FLINK-6109 Project: Flink
Issue Type: New Feature
Components: Kafka Connector, Streaming Connectors
Reporter: Tzu-Li (Gordon) Tai
This is a feature discussed in this ML:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Telling-if-a-job-has-caught-up-with-Kafka-td12261.html.
As discussed, we can expose two kinds of "consumer lag" metrics for this:
- "current consumer lag for partition": the current difference between the latest offset and the last collected record record of a partition. This metric is calculated and updated at a configurable interval. This metric basically serves as an indicator of how the consumer is keeping up with the head of partitions. I propose to name this `currentOffsetLag`.
- "Consumer lag of last checkpoint": the difference between the latest offset and the offset stored in the checkpoint of a partition. This metric is only updated when checkpoints are completed. It serves as an indicator of how much data may need to be replayed in case of a failure. I propose to name this `lastCheckpointedOffsetLag`.
The granularity of the metric is per-FlinkKafkaConsumer, and independent of the consumer group.id used (the offset used to calculate consumer lag is the internal offset state of the FlinkKafkaConsumer, not the consumer group's committed offsets in Kafka).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Locked
