[jira] [Created] (FLINK-14039) Flink Kinesis consumer: configurable per-shard consumption rate when running in adaptive mode

[Shang Yuanchun (Jira)]

Ying Xu created FLINK-14039:
-------------------------------

             Summary: Flink Kinesis consumer: configurable per-shard consumption rate when running in adaptive mode
                 Key: FLINK-14039
                 URL: https://issues.apache.org/jira/browse/FLINK-14039
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kinesis
            Reporter: Ying Xu


Currently, Flink kinesis connector has a fixed [2MB|https://github.com/apache/flink/blob/78748ea1aee8f9d0c0499180a2ef455490b32b24/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java#L59-L61] target rate (per-shard) when running in adaptive rate mode.  In specific scenarios, it is desirable that users would want a different target rate. For example, when two Kinesis consumers share a common stream, the user may want to de-prioritize one stream such that it runs with a target rate < 2MB. 

It is relatively straightforward to implement this feature – simply add a per-shard target rate consumer config and has the default set to 2MB. 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)