Danny Cranmer created FLINK-20629:
-------------------------------------
Summary: [Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary
Key: FLINK-20629
URL:
https://issues.apache.org/jira/browse/FLINK-20629 Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Affects Versions: 1.12.0
Reporter: Danny Cranmer
Fix For: 1.12.1
*Background*
The Kinesis EFO connector invokes {{DescribeStream}} during startup to acquire the stream ARN. This call also includes the shard information and has a TPS of 10. A similar service exists, {{DescribeStreamSummary}} that has a TPS of 20 and a lighter response payload size.
During startup sources with high parallelism compete to call this service (in {{LAZY}} mode), resulting in backoff and retry. Essentially the startup time can grow by 1s for every 10 parallelism, due to the 10 TPS. Migrating to {{DescribeStreamSummary}} will improve startup time.
*Scope*
Migrate call to {{DescribeStream}} to use {{DescribeStreamSummary}} instead.
*Note*
I have targeted {{1.12.1}}, let me know if we should instead target {{1.13}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
