[jira] [Created] (FLINK-2004) Memory leack in presence of failed checkpoints in KafkaSource

[Shang Yuanchun (Jira)]

Stephan Ewen created FLINK-2004:
-----------------------------------

             Summary: Memory leack in presence of failed checkpoints in KafkaSource
                 Key: FLINK-2004
                 URL: https://issues.apache.org/jira/browse/FLINK-2004
             Project: Flink
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 0.9
            Reporter: Stephan Ewen
            Assignee: Robert Metzger
            Priority: Critical
             Fix For: 0.9


Checkpoints that fail never send a commit message to the tasks.

Maintaining a map of all pending checkpoints introduces a memory leak, as entries for failed checkpoints will never be removed.

Approaches to fix this:
  - The source cleans up entries from older checkpoints once a checkpoint is committed (simple implementation in a linked hash map)
  - The commit message could include the optional state handle (source needs not maintain the map)
  - The checkpoint coordinator could send messages for failed checkpoints?





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)