[jira] [Created] (FLINK-8542) Do not indefinitely store closed shard's state in the FlinkKinesisConsumer

[Shang Yuanchun (Jira)]

Tzu-Li (Gordon) Tai created FLINK-8542:
------------------------------------------

             Summary: Do not indefinitely store closed shard's state in the FlinkKinesisConsumer
                 Key: FLINK-8542
                 URL: https://issues.apache.org/jira/browse/FLINK-8542
             Project: Flink
          Issue Type: Improvement
          Components: Kinesis Connector
            Reporter: Tzu-Li (Gordon) Tai


See original discussion here: [https://github.com/apache/flink/pull/5337|https://github.com/apache/flink/pull/5337#issuecomment-362227711]

Currently, the Kinesis consumer keeps a list of {{(StreamShardMetadata, SequenceNumber)}} as its state. That list also contains all shards that have been closed already, and is kept in the state indefinitely so that on restore, we know that a closed shard is already fully consumed,

The downside of this, is that the state size of the Kinesis consumer can basically grow without bounds, as the consumed Kinesis streams are resharded and more and more closed shards are present.

Some possible solutions have been discussed in the linked PR comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)