[Discuss] FLIP-43: Savepoint Connector

[Seth Wiesman-4]

Hey Everyone!

Gordon and I have been discussing adding a savepoint connector to flink for reading, writing, and modifying savepoints.

This is useful for:
* Analyzing state for interesting patterns
* Troubleshooting or auditing jobs by checking for discrepancies in state
* Bootstrapping state for new applications
* Modifying savepoints to make breaking scheme changes or correct invalid state

We are looking forwards to your feedback!

This is the FLIP:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector

Seth

[Piotr Nowojski-3]

I was long awaiting this feature! I can not help much with reviewing, but big +1 from my side :)

One thing that would be great for analyzing the state and possible smaller modifications, would be to hook this in with Flink SQL :) Especially if it could be done in a way that would work out of the box with minimal or no configuration required, like just opening SQL CLI with just maybe a pointer to savepoint id?

Side question, are you aiming with this for 1.9 release? 1.10? Or you do not have a specific time frame? I’m asking since users are quite frequently asking for this and it would be good to know what to answer them.

Piotrek

> On 29 May 2019, at 17:51, Seth Wiesman <[hidden email]> wrote:
>
> Hey Everyone!
>
> Gordon and I have been discussing adding a savepoint connector to flink for reading, writing, and modifying savepoints.
>
> This is useful for:
> * Analyzing state for interesting patterns
> * Troubleshooting or auditing jobs by checking for discrepancies in state
> * Bootstrapping state for new applications
> * Modifying savepoints to make breaking scheme changes or correct invalid state
>
> We are looking forwards to your feedback!
>
> This is the FLIP:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector
>
> Seth

[Tzu-Li (Gordon) Tai]

@Piotr
Yes, we're aiming this for the 1.9 release. This was also mentioned in the
recent 1.9 feature discussion thread [1].

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html

On Fri, May 31, 2019 at 4:34 PM Piotr Nowojski <[hidden email]> wrote:

> I was long awaiting this feature! I can not help much with reviewing, but
> big +1 from my side :)
>
> One thing that would be great for analyzing the state and possible smaller
> modifications, would be to hook this in with Flink SQL :) Especially if it
> could be done in a way that would work out of the box with minimal or no
> configuration required, like just opening SQL CLI with just maybe a pointer
> to savepoint id?
>
> Side question, are you aiming with this for 1.9 release? 1.10? Or you do
> not have a specific time frame? I’m asking since users are quite frequently
> asking for this and it would be good to know what to answer them.
>
> Piotrek
>
> > On 29 May 2019, at 17:51, Seth Wiesman <[hidden email]> wrote:
> >
> > Hey Everyone!
> >
> > Gordon and I have been discussing adding a savepoint connector to flink
> for reading, writing, and modifying savepoints.
> >
> > This is useful for:
> > * Analyzing state for interesting patterns
> > * Troubleshooting or auditing jobs by checking for discrepancies in state
> > * Bootstrapping state for new applications
> > * Modifying savepoints to make breaking scheme changes or correct
> invalid state
> >
> > We are looking forwards to your feedback!
> >
> > This is the FLIP:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector
> >
> > Seth
>
>

[Seth Wiesman-4]

@Piotr

I definitely would like to see this have sql integrations at some point. The reason for holding off is that to do so would require savepoint format, it is not currently possible to discover states and schemas without state descriptors in a robust way.

I think it’s best to keep this initial implementation focused and add those changes if there is adoption and interest in the community.

Seth

> On May 31, 2019, at 4:39 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:
>
> @Piotr
> Yes, we're aiming this for the 1.9 release. This was also mentioned in the
> recent 1.9 feature discussion thread [1].
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html
>
>> On Fri, May 31, 2019 at 4:34 PM Piotr Nowojski <[hidden email]> wrote:
>>
>> I was long awaiting this feature! I can not help much with reviewing, but
>> big +1 from my side :)
>>
>> One thing that would be great for analyzing the state and possible smaller
>> modifications, would be to hook this in with Flink SQL :) Especially if it
>> could be done in a way that would work out of the box with minimal or no
>> configuration required, like just opening SQL CLI with just maybe a pointer
>> to savepoint id?
>>
>> Side question, are you aiming with this for 1.9 release? 1.10? Or you do
>> not have a specific time frame? I’m asking since users are quite frequently
>> asking for this and it would be good to know what to answer them.
>>
>> Piotrek
>>
>>> On 29 May 2019, at 17:51, Seth Wiesman <[hidden email]> wrote:
>>>
>>> Hey Everyone!
>>>
>>> Gordon and I have been discussing adding a savepoint connector to flink
>> for reading, writing, and modifying savepoints.
>>>
>>> This is useful for:
>>> * Analyzing state for interesting patterns
>>> * Troubleshooting or auditing jobs by checking for discrepancies in state
>>> * Bootstrapping state for new applications
>>> * Modifying savepoints to make breaking scheme changes or correct
>> invalid state
>>>
>>> We are looking forwards to your feedback!
>>>
>>> This is the FLIP:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector
>>>
>>> Seth
>>
>>

[Seth Wiesman-4]

And I can definitely imagine a “savepoint catalog” at some point in the future.

Seth

> On May 31, 2019, at 4:39 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:
>
> @Piotr
> Yes, we're aiming this for the 1.9 release. This was also mentioned in the
> recent 1.9 feature discussion thread [1].
>
> [1]
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html
>
>> On Fri, May 31, 2019 at 4:34 PM Piotr Nowojski <[hidden email]> wrote:
>>
>> I was long awaiting this feature! I can not help much with reviewing, but
>> big +1 from my side :)
>>
>> One thing that would be great for analyzing the state and possible smaller
>> modifications, would be to hook this in with Flink SQL :) Especially if it
>> could be done in a way that would work out of the box with minimal or no
>> configuration required, like just opening SQL CLI with just maybe a pointer
>> to savepoint id?
>>
>> Side question, are you aiming with this for 1.9 release? 1.10? Or you do
>> not have a specific time frame? I’m asking since users are quite frequently
>> asking for this and it would be good to know what to answer them.
>>
>> Piotrek
>>
>>> On 29 May 2019, at 17:51, Seth Wiesman <[hidden email]> wrote:
>>>
>>> Hey Everyone!
>>>
>>> Gordon and I have been discussing adding a savepoint connector to flink
>> for reading, writing, and modifying savepoints.
>>>
>>> This is useful for:
>>> * Analyzing state for interesting patterns
>>> * Troubleshooting or auditing jobs by checking for discrepancies in state
>>> * Bootstrapping state for new applications
>>> * Modifying savepoints to make breaking scheme changes or correct
>> invalid state
>>>
>>> We are looking forwards to your feedback!
>>>
>>> This is the FLIP:
>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector
>>>
>>> Seth
>>
>>

[Piotr Nowojski-3]

> I think it’s best to keep this initial implementation focused and add those changes if there is adoption and interest in the community.

I agree. I didn’t mean to hold the implementation/acceptance of this until someone solve the SQL story :)

Piotrek

> On 31 May 2019, at 13:18, Seth Wiesman <[hidden email]> wrote:
>
> And I can definitely imagine a “savepoint catalog” at some point in the future.
>
> Seth
>
>> On May 31, 2019, at 4:39 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:
>>
>> @Piotr
>> Yes, we're aiming this for the 1.9 release. This was also mentioned in the
>> recent 1.9 feature discussion thread [1].
>>
>> [1]
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Features-for-Apache-Flink-1-9-0-td28701.html
>>
>>> On Fri, May 31, 2019 at 4:34 PM Piotr Nowojski <[hidden email]> wrote:
>>>
>>> I was long awaiting this feature! I can not help much with reviewing, but
>>> big +1 from my side :)
>>>
>>> One thing that would be great for analyzing the state and possible smaller
>>> modifications, would be to hook this in with Flink SQL :) Especially if it
>>> could be done in a way that would work out of the box with minimal or no
>>> configuration required, like just opening SQL CLI with just maybe a pointer
>>> to savepoint id?
>>>
>>> Side question, are you aiming with this for 1.9 release? 1.10? Or you do
>>> not have a specific time frame? I’m asking since users are quite frequently
>>> asking for this and it would be good to know what to answer them.
>>>
>>> Piotrek
>>>
>>>> On 29 May 2019, at 17:51, Seth Wiesman <[hidden email]> wrote:
>>>>
>>>> Hey Everyone!
>>>>
>>>> Gordon and I have been discussing adding a savepoint connector to flink
>>> for reading, writing, and modifying savepoints.
>>>>
>>>> This is useful for:
>>>> * Analyzing state for interesting patterns
>>>> * Troubleshooting or auditing jobs by checking for discrepancies in state
>>>> * Bootstrapping state for new applications
>>>> * Modifying savepoints to make breaking scheme changes or correct
>>> invalid state
>>>>
>>>> We are looking forwards to your feedback!
>>>>
>>>> This is the FLIP:
>>>>
>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector
>>>>
>>>> Seth
>>>
>>>