[jira] [Created] (FLINK-11501) Add a ratelimiting feature to the FlinkKafkaConsumer

[Shang Yuanchun (Jira)]

Lakshmi Rao created FLINK-11501:
-----------------------------------

             Summary: Add a ratelimiting feature to the FlinkKafkaConsumer
                 Key: FLINK-11501
                 URL: https://issues.apache.org/jira/browse/FLINK-11501
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
            Reporter: Lakshmi Rao
            Assignee: Lakshmi Rao


There are instances when a Flink job that reads from Kafka can read at a significantly high throughput (particularly while processing a backlog) and degrade the underlying Kafka cluster.

While Kafka quotas are perhaps the best way to enforce this ratelimiting, there are cases where such a setup is not available or easily enabled. 

The approach is essentially to use Guava's rate limiter to ratelimit the bytes read from Kafka.

More discussion here: https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)