Add relative path support in Savepoint Connector

[Jiayi Liao]

Hi all,
    Firstly I appreciate Gordon and Seth’s effort on this feature, which is really helpful to our production use. Like you mentioned in the FLINK-12047, one of the production uses is that we use the existing state to derive new state. However, since the state handle is using the absolute path to get the input stream, we need to directly operate the state in production environment, which is not an anxiety-reducing situation, at least for me.
    So I wonder if we can add the relative path support in this module because the files are persisted in a directory after we take a savepoint, which makes it achievable. I’m not sure whether my scenario is a common case or not, but I think I can give my contributions if you all are okay about this.




Best Regards,
Jiayi Liao

[Jiayi Liao]

Hi again,
Anyone has any opinion on this topic?


Best Regards,
Jiayi Liao


Original Message
Sender:[hidden email]
Recipient:[hidden email]
Cc:Tzu-Li (Gordon) [hidden email]
Date:Tuesday, Jul 16, 2019 15:24
Subject:Add relative path support in Savepoint Connector


Hi all,
    Firstly I appreciate Gordon and Seth’s effort on this feature, which is really helpful to our production use. Like you mentioned in the FLINK-12047, one of the production uses is that we use the existing state to derive new state. However, since the state handle is using the absolute path to get the input stream, we need to directly operate the state in production environment, which is not an anxiety-reducing situation, at least for me.
    So I wonder if we can add the relative path support in this module because the files are persisted in a directory after we take a savepoint, which makes it achievable. I’m not sure whether my scenario is a common case or not, but I think I can give my contributions if you all are okay about this.




Best Regards,
Jiayi Liao

[Konstantin Knauf-3]

Hi Jiayi,

I think, this is not an issue with the State Processor API specifically,
but with savepoints in general. The _metadata file of a savepoint uses
absolute path references. There is a pretty old Jira ticket, which already
mentioned this limitation [1].

Stefan (cc) might know more about any ongoing development in that direction
and might have an idea about the effort of making savepoints relocatable.

Best,

Konstantin

[1] https://issues.apache.org/jira/browse/FLINK-5763

On Wed, Jul 17, 2019 at 8:35 AM bupt_ljy <[hidden email]> wrote:

> Hi again,
> Anyone has any opinion on this topic?
>
>
> Best Regards,
> Jiayi Liao
>
>
> Original Message
> Sender:[hidden email]
> Recipient:[hidden email]
> Cc:Tzu-Li (Gordon) [hidden email]
> Date:Tuesday, Jul 16, 2019 15:24
> Subject:Add relative path support in Savepoint Connector
>
>
> Hi all,
>     Firstly I appreciate Gordon and Seth’s effort on this feature, which
> is really helpful to our production use. Like you mentioned in the
> FLINK-12047, one of the production uses is that we use the existing state
> to derive new state. However, since the state handle is using the absolute
> path to get the input stream, we need to directly operate the state in
> production environment, which is not an anxiety-reducing situation, at
> least for me.
>     So I wonder if we can add the relative path support in this module
> because the files are persisted in a directory after we take a savepoint,
> which makes it achievable. I’m not sure whether my scenario is a common
> case or not, but I think I can give my contributions if you all are okay
> about this.
>
>
>
>
> Best Regards,
> Jiayi Liao

--

Konstantin Knauf | Solutions Architect

[phone]


Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010


--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

[Jiayi Liao]

Hi Konstantin,
Thank you for your feedback. You’re right that this part belongs to the savepoint desrializing.
    This is an old issue which should be resolve before 1.3 version according to the comments. Anyway, I’m going to keep following this.


Best Regards,
Jiayi Liao


Original Message
Sender:Konstantin [hidden email]
Recipient:[hidden email]; Stefan [hidden email]
Date:Wednesday, Jul 17, 2019 16:28
Subject:Re: Add relative path support in Savepoint Connector


Hi Jiayi, I think, this is not an issue with the State Processor API specifically, but with savepoints in general. The _metadata file of a savepoint uses absolute path references. There is a pretty old Jira ticket, which already mentioned this limitation [1]. Stefan (cc) might know more about any ongoing development in that direction and might have an idea about the effort of making savepoints relocatable. Best, Konstantin [1] https://issues.apache.org/jira/browse/FLINK-5763 On Wed, Jul 17, 2019 at 8:35 AM bupt_ljy [hidden email] wrote:  Hi again,  Anyone has any opinion on this topic?    Best Regards,  Jiayi Liao    Original Message  Sender:[hidden email]  Recipient:[hidden email]  Cc:Tzu-Li (Gordon) [hidden email]  Date:Tuesday, Jul 16, 2019 15:24  Subject:Add relative path support in Savepoint Connector    Hi all,  Firstly I appreciate Gordon and Seth’s effort on this feature, which  is really helpful to our production use. Like you mentioned in the  FLINK-12047, one of the production uses is that we use the existing state  to derive new state. However, since the state handle is using the absolute  path to get the input stream, we need to directly operate the state in  production environment, which is not an anxiety-reducing situation, at  least for me.  So I wonder if we can add the relative path support in this module  because the files are persisted in a directory after we take a savepoint,  which makes it achievable. I’m not sure whether my scenario is a common  case or not, but I think I can give my contributions if you all are okay  about this.      Best Regards,  Jiayi Liao -- Konstantin Knauf | Solutions Architect [phone] Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010 -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen