Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

[LakeShen]

Hi community，
        now I have a flink sql job, and I set the flink sql sate retention
time, there are three dir in flink checkpoint dir :
1. chk -xx dir
2. shared dir
3. taskowned dir

I find the shared dir store the last year checkpoint state,the only reason
I thought is that the latest
checkpoint retain reference of last year checkpoint state file.
Are there any other reason to lead this? Or is it a bug?

Thanks to your replay.

Best wishes,
Lake Shen

[Yun Tang]

Hi Lake

A more suitable place for this mail should be in user-mail list.

There are three reasons why this could happen:

  1.  This file is orphan file e.g. file uploaded during one checkpoint but task manager exited unexpectedly leave that checkpoint not completed.
  2.  This file should be removed by checkpoint coordinator but takes too long to complete before job shut down.
  3.  This file is still useful. This is possible in theory because some specific rocksDB sst file might not be selected during compactions for a long time.

Best
Yun Tang
________________________________
From: LakeShen <[hidden email]>
Sent: Sunday, January 19, 2020 18:55
To: user <[hidden email]>; [hidden email] <[hidden email]>; dev <[hidden email]>
Subject: Flink 1.6, increment Checkpoint, the shared dir stored the last year checkpoint state

Hi community，
        now I have a flink sql job, and I set the flink sql sate retention time, there are three dir in flink checkpoint dir :
1. chk -xx dir
2. shared dir
3. taskowned dir

I find the shared dir store the last year checkpoint state,the only reason I thought is that the latest
checkpoint retain reference of last year checkpoint state file.
Are there any other reason to lead this? Or is it a bug?

Thanks to your replay.

Best wishes,
Lake Shen