[jira] [Created] (FLINK-21413) TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend

[Shang Yuanchun (Jira)]

Jiayi Liao created FLINK-21413:
----------------------------------

             Summary: TtlMapState and TtlListState cannot be clean completely with Filesystem StateBackend
                 Key: FLINK-21413
                 URL: https://issues.apache.org/jira/browse/FLINK-21413
             Project: Flink
          Issue Type: Bug
          Components: Runtime / State Backends
    Affects Versions: 1.9.0
            Reporter: Jiayi Liao


Take the #TtlMapState as an example,

 
{code:java}
public Map<UK, TtlValue<UV>> getUnexpiredOrNull(@Nonnull Map<UK, TtlValue<UV>> ttlValue) {
        Map<UK, TtlValue<UV>> unexpired = new HashMap<>();
        TypeSerializer<TtlValue<UV>> valueSerializer =
                ((MapSerializer<UK, TtlValue<UV>>) original.getValueSerializer()).getValueSerializer();
        for (Map.Entry<UK, TtlValue<UV>> e : ttlValue.entrySet()) {
                if (!expired(e.getValue())) {
                        // we have to do the defensive copy to update the value
                        unexpired.put(e.getKey(), valueSerializer.copy(e.getValue()));
                }
        }
        return ttlValue.size() == unexpired.size() ? ttlValue : unexpired;
}
{code}
 

The returned value will never be null and the #StateEntry will exists forever, which leads to memory leak if the key's range of the stream is very large. Below we can see that 20+ millison uncleared TtlStateMap could take up several GB memory.

 

!image-2021-02-19-11-13-38-691.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)