Tzu-Li (Gordon) Tai created FLINK-7622:
------------------------------------------
Summary: Respect local KPL queue size in FlinkKinesisProducer when adding records to KPL client
Key: FLINK-7622
URL:
https://issues.apache.org/jira/browse/FLINK-7622 Project: Flink
Issue Type: Improvement
Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
This issue was brought to discussion by [~sthm] offline.
Currently, records are added to the Kinesis KPL producer client without checking the number of outstanding records within the local KPL queue. This manner is basically neglecting backpressure when producing to Kinesis through KPL, and can therefore exhaust system resources.
We should respect {{producer.getOutstandingRecordsCount()}} as a measure of backpressure, and propagate backpressure upstream by blocking further sink invocations when some threshold of outstanding record count is exceeded. The recommended threshold [1] seems to be 10,000.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Locked
