[DISCUSS] FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers

[Cranmer, Danny]

Hello everyone,

This is a discussion thread for the FLIP I have recently created:

  *   FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers<https://cwiki.apache.org/confluence/display/FLINK/FLIP-128%3A+Enhanced+Fan+Out+for+AWS+Kinesis+Consumers>
  *   https://issues.apache.org/jira/browse/FLINK-17688

Enhanced Fan Out (EFO) allows AWS Kinesis Data Stream (KDS) consumers to utilise a dedicated read throughput, rather than a shared quota. HTTP/2 reduces latency and typically gives a 65% performance boost [1]. EFO is not currently supported by the Flink Kinesis Consumer. Adding EFO support would allow Flink applications to reap the benefits, widening Flink adoption. Existing applications will be able to optionally perform a backwards compatible library upgrade and configuration tweak to inherit the performance benefits.

[1] https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/

I look forward to your feedback,

Thanks,