[jira] [Created] (FLINK-17583) Allow option to store a savepoint's _metadata file separate from its data files

[Shang Yuanchun (Jira)]

Steve Bairos created FLINK-17583:
------------------------------------

             Summary: Allow option to store a savepoint's _metadata file separate from its data files
                 Key: FLINK-17583
                 URL: https://issues.apache.org/jira/browse/FLINK-17583
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Checkpointing
    Affects Versions: 1.9.1
            Reporter: Steve Bairos


(In the description I mainly talk about savepoints, but the plan )

We have a deployment framework that often needs to be able to return a list of valid savepoints in S3 with a certain prefix. Our assertion is that if an S3 object ends with '_metadata', then it is a valid savepoint. In order to generate the list of valid savepoints, we need to locate all of the _metadata files that start with a given prefix.

For example, if our S3 bucket's paths look like this:

 
{code:java}
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-1a2b3c4d5e/_metadata
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-1a2b3c4d5e/9c165546-c326-43c0-9f47-f9a2cfd000ed
... thousands of other savepoint data files
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-1a2b3c4d5e/9c757e5b-92b7-47b8-bfe8-cfe70eb28702
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-9999999999/_metadata
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-9999999999/41297fd5-40df-4683-bfb6-534bfddae92a
... thousands of other savepoint data files
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-9999999999/acbe839a-1ec7-4b41-9d87-595d557c2ac6
s3://bucket/savepoints/my-job1/2020-04-02/savepoint-987654-1100110011/_metadata
s3://bucket/savepoints/my-job1/2020-04-02/savepoint-987654-1100110011/2d2f5551-56a7-4fea-b25b-b0156660c650
.... thousands of other savepoint data files
s3://bucket/savepoints/my-job1/2020-04-02/savepoint-987654-1100110011/c8c410df-5fb0-46a0-84c5-43e1575e8dc5
... dozens of other savepoint dirs


{code}
 

In order to get a list of all savepoints that my-job1 could possibly start with, we would want to get all the savepoints that start with the prefix:
{code:java}
s3://bucket/savepoints/my-job1 {code}
Ideally, we would want to have the ability to get a list like this from S3:
{code:java}
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-1a2b3c4d5e/_metadata
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-9999999999/_metadata
s3://bucket/savepoints/my-job1/2020-04-02/savepoint-987654-1100110011/_metadata{code}
Unfortunately there is no easy way to get this value because S3's API only allows you to search based on prefix and not postfix. Listing all objects with the prefix 's3://bucket/savepoints/my-job1' and then filtering the list to only include the files that contain _metadata will also not work because there are thousands of savepoint data files that have the same prefix such as:
{code:java}
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-1a2b3c4d5e/9c165546-c326-43c0-9f47-f9a2cfd000ed
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-1a2b3c4d5e/9c757e5b-92b7-47b8-bfe8-cfe70eb28702
s3://bucket/savepoints/my-job1/2020-04-01/savepoint-123456-9999999999/acbe839a-1ec7-4b41-9d87-595d557c2ac6
etc.{code}
 

I propose that we add a configuration in a similar vein to the S3 entropy injector which allows us to store the _metadata file in a separate path from the savepoint's data files. For example, with this hypothetical configuration:
{code:java}
state.checkpoints.split.key: _datasplit_
state.checkpoints.split.metadata.dir: metadata
state.checkpoints.split.data.dir: data{code}
When a user triggers a savepoint with the path
{code:java}
s3://bucket/savepoints/_datasplit_/my-job1/2020-05-07/ {code}
The resulting savepoint that is created looks like:
{code:java}
s3://bucket/savepoints/metadata/my-job1/2020-05-07/savepoint-654321-abcdef9876/_metadata
s3://bucket/savepoints/data/my-job1/2020-05-07/savepoint-654321-abcdef9876/a50fc483-3581-4b55-a37e-b7c61b3ee47f
s3://bucket/savepoints/data/my-job1/2020-05-07/savepoint-654321-abcdef9876/b0c6b7c0-6b94-43ae-8678-2f7640af1523
s3://bucket/savepoints/data/my-job1/2020-05-07/savepoint-654321-abcdef9876/c1855b35-c0b7-4347-9352-88423998e5ec{code}
Notice that the metadata's prefix is 
{code:java}
 s3://bucket/savepoints/metadata/my-job1/2020-05-07/{code}
and the data files' prefix is
{code:java}
 s3://bucket/savepoints/data/my-job1/2020-05-07/{code}
That way if I want to list all the savepoints for my-job1, I can just list all the objects with the prefix 
{code:java}
 aws s3 ls --recursive s3://bucket/savepoints/metadata/my-job1/{code}
And I can get a clean list of just the _metadata files easily.

 

One alternative that we've thought about is using is the entropy injection. It technically does separate the _metadata file from the rest of the data as well but it kind of makes a mess of entropy dirs in S3 so it's not our ideal choice. 

 

I'm happy to take a shot at implementing the solution I suggested if it is an acceptable solution for Flink. 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)