[jira] [Created] (FLINK-6215) Make the StatefulSequenceSource scalable.

[Shang Yuanchun (Jira)]

Kostas Kloudas created FLINK-6215:
-------------------------------------

             Summary: Make the StatefulSequenceSource scalable.
                 Key: FLINK-6215
                 URL: https://issues.apache.org/jira/browse/FLINK-6215
             Project: Flink
          Issue Type: Bug
          Components: DataStream API
    Affects Versions: 1.3.0
            Reporter: Kostas Kloudas
             Fix For: 1.3.0


Currently the {{StatefulSequenceSource}} instantiates all the elements to emit first and keeps them in memory. This is not scalable as for large sequences of elements this can lead to out of memory exceptions.

To solve this, we can pre-partition the sequence of elements based on the {{maxParallelism}} parameter, and just keep state (to checkpoint) per such partition.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)