Savepoints and memory statebackend

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Savepoints and memory statebackend

Gyula Fóra
Hey!

The flink docs on savepoints say that they cannot be used in combination
with the memory statebackend as the savepoint backend will only store
pointers to the state.


If i understand correctly these pointers are the StateHandles that are
stored by the JobManager. In case of the Memory state backend the state is
stored serialized in the handle so I would assume it should work with
savepoints as well.

Am I missing something?

Cheers,
Gyula
Reply | Threaded
Open this post in threaded view
|

Re: Savepoints and memory statebackend

Aljoscha Krettek-2
Hi,
theoretically, it could work (if I'm not missing anything). But the
savepoint would be lost as soon as the JobManager is shut down.

Cheers,
Aljoscha

On Fri, 27 May 2016 at 18:09 Gyula Fóra <[hidden email]> wrote:

> Hey!
>
> The flink docs on savepoints say that they cannot be used in combination
> with the memory statebackend as the savepoint backend will only store
> pointers to the state.
>
>
> If i understand correctly these pointers are the StateHandles that are
> stored by the JobManager. In case of the Memory state backend the state is
> stored serialized in the handle so I would assume it should work with
> savepoints as well.
>
> Am I missing something?
>
> Cheers,
> Gyula
>
Reply | Threaded
Open this post in threaded view
|

Re: Savepoints and memory statebackend

Gyula Fóra
Not if the savepoint backend is Filesystem, right? So it shouldnt really be
a matter of the statebackend, only the savepoint backend.

Gyula

On Fri, May 27, 2016, 18:47 Aljoscha Krettek <[hidden email]> wrote:

> Hi,
> theoretically, it could work (if I'm not missing anything). But the
> savepoint would be lost as soon as the JobManager is shut down.
>
> Cheers,
> Aljoscha
>
> On Fri, 27 May 2016 at 18:09 Gyula Fóra <[hidden email]> wrote:
>
> > Hey!
> >
> > The flink docs on savepoints say that they cannot be used in combination
> > with the memory statebackend as the savepoint backend will only store
> > pointers to the state.
> >
> >
> > If i understand correctly these pointers are the StateHandles that are
> > stored by the JobManager. In case of the Memory state backend the state
> is
> > stored serialized in the handle so I would assume it should work with
> > savepoints as well.
> >
> > Am I missing something?
> >
> > Cheers,
> > Gyula
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Savepoints and memory statebackend

Aljoscha Krettek-2
Not sure about that. Maybe Ufuk can chime in.

On Fri, 27 May 2016 at 19:40 Gyula Fóra <[hidden email]> wrote:

> Not if the savepoint backend is Filesystem, right? So it shouldnt really be
> a matter of the statebackend, only the savepoint backend.
>
> Gyula
>
> On Fri, May 27, 2016, 18:47 Aljoscha Krettek <[hidden email]> wrote:
>
> > Hi,
> > theoretically, it could work (if I'm not missing anything). But the
> > savepoint would be lost as soon as the JobManager is shut down.
> >
> > Cheers,
> > Aljoscha
> >
> > On Fri, 27 May 2016 at 18:09 Gyula Fóra <[hidden email]> wrote:
> >
> > > Hey!
> > >
> > > The flink docs on savepoints say that they cannot be used in
> combination
> > > with the memory statebackend as the savepoint backend will only store
> > > pointers to the state.
> > >
> > >
> > > If i understand correctly these pointers are the StateHandles that are
> > > stored by the JobManager. In case of the Memory state backend the state
> > is
> > > stored serialized in the handle so I would assume it should work with
> > > savepoints as well.
> > >
> > > Am I missing something?
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Savepoints and memory statebackend

Ufuk Celebi-2
Hey Gyula!

You are right that in this case the memory snapshots go to the job
manager and are part of the save point. The docs seem to be off there.

The whole save point backend and pointer business should be removed
though in favour of making save points self contained and always go to
files. I will start a discussion about this for the next release.

– Ufuk





On Fri, May 27, 2016 at 8:04 PM, Aljoscha Krettek <[hidden email]> wrote:

> Not sure about that. Maybe Ufuk can chime in.
>
>
> On Fri, 27 May 2016 at 19:40 Gyula Fóra <[hidden email]> wrote:
>>
>> Not if the savepoint backend is Filesystem, right? So it shouldnt really
>> be
>> a matter of the statebackend, only the savepoint backend.
>>
>> Gyula
>>
>> On Fri, May 27, 2016, 18:47 Aljoscha Krettek <[hidden email]> wrote:
>>
>> > Hi,
>> > theoretically, it could work (if I'm not missing anything). But the
>> > savepoint would be lost as soon as the JobManager is shut down.
>> >
>> > Cheers,
>> > Aljoscha
>> >
>> > On Fri, 27 May 2016 at 18:09 Gyula Fóra <[hidden email]> wrote:
>> >
>> > > Hey!
>> > >
>> > > The flink docs on savepoints say that they cannot be used in
>> > > combination
>> > > with the memory statebackend as the savepoint backend will only store
>> > > pointers to the state.
>> > >
>> > >
>> > > If i understand correctly these pointers are the StateHandles that are
>> > > stored by the JobManager. In case of the Memory state backend the
>> > > state
>> > is
>> > > stored serialized in the handle so I would assume it should work with
>> > > savepoints as well.
>> > >
>> > > Am I missing something?
>> > >
>> > > Cheers,
>> > > Gyula
>> > >
>> >