Incremental checkpoint branch

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Incremental checkpoint branch

Vishnu Viswanath
Hi,

Can someone point me to the branch where the ongoing work for incremental
checkpoint is going on, I would like to try it out even if the work is not
complete.

I have a use case where the state size increase about ~1gb every 5 minutes.

Thanks,
Vishnu
Reply | Threaded
Open this post in threaded view
|

Re: Incremental checkpoint branch

Shaoxuan Wang
Vinshnu,
You can find the latest design discussion for incremental checkpoint in
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-Flink-td15931.html
 @Stefan Richter
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter>
and @Xiaogang
Shi
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=xiaogang.shi>
might
be the right person to provide you its on-going status.

cheers,
Shaoxuan

On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath <
[hidden email]> wrote:

> Hi,
>
> Can someone point me to the branch where the ongoing work for incremental
> checkpoint is going on, I would like to try it out even if the work is not
> complete.
>
> I have a use case where the state size increase about ~1gb every 5 minutes.
>
> Thanks,
> Vishnu
>
Reply | Threaded
Open this post in threaded view
|

Re: Incremental checkpoint branch

SHI Xiaogang
Hi Vinshnu,

We have obtained an initial design of incremental checkpointing [1] and
will start working on incremental checkpointing the next week. You can
watch the issue  FLINK-5053 [2] to get timely notification of the updates.
All suggestions are welcome.

[1]
https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jSUMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z
[2] https://issues.apache.org/jira/browse/FLINK-5053

Regards,
Xiaogang



2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <[hidden email]>:

> Vinshnu,
> You can find the latest design discussion for incremental checkpoint in
> http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-Flink-td15931.html
>  @Stefan Richter
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter>
> and @Xiaogang
> Shi
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=xiaogang.shi>
> might
> be the right person to provide you its on-going status.
>
> cheers,
> Shaoxuan
>
> On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath <
> [hidden email]> wrote:
>
> > Hi,
> >
> > Can someone point me to the branch where the ongoing work for incremental
> > checkpoint is going on, I would like to try it out even if the work is
> not
> > complete.
> >
> > I have a use case where the state size increase about ~1gb every 5
> minutes.
> >
> > Thanks,
> > Vishnu
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Incremental checkpoint branch

Vishnu Viswanath
Thanks everyone for the reply.

@Xiaogang design document looks very good, looking forward to check the
implementation and to try it out. I see that hdfs:// is mentioned as the
backend storage in the documentation, what about s3://. Is it something
that needed to be considered in the design, or is it handled by RocksDB.

Thanks,
Vishnu

On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <[hidden email]> wrote:

> Hi Vinshnu,
>
> We have obtained an initial design of incremental checkpointing [1] and
> will start working on incremental checkpointing the next week. You can
> watch the issue  FLINK-5053 [2] to get timely notification of the updates.
> All suggestions are welcome.
>
> [1]
> https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS
> UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z
> [2] https://issues.apache.org/jira/browse/FLINK-5053
>
> Regards,
> Xiaogang
>
>
>
> 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <[hidden email]>:
>
> > Vinshnu,
> > You can find the latest design discussion for incremental checkpoint in
> > http://apache-flink-mailing-list-archive.1008284.n3.
> > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-
> Flink-td15931.html
> >  @Stefan Richter
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter>
> > and @Xiaogang
> > Shi
> > <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> name=xiaogang.shi>
> > might
> > be the right person to provide you its on-going status.
> >
> > cheers,
> > Shaoxuan
> >
> > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath <
> > [hidden email]> wrote:
> >
> > > Hi,
> > >
> > > Can someone point me to the branch where the ongoing work for
> incremental
> > > checkpoint is going on, I would like to try it out even if the work is
> > not
> > > complete.
> > >
> > > I have a use case where the state size increase about ~1gb every 5
> > minutes.
> > >
> > > Thanks,
> > > Vishnu
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Incremental checkpoint branch

Stephan Ewen
Hi!

The concrete storage location (HDFS / S3) should be independent of the
imeplementation. The design should work for both.

Stephan


On Mon, Mar 6, 2017 at 5:38 PM, Vishnu Viswanath <
[hidden email]> wrote:

> Thanks everyone for the reply.
>
> @Xiaogang design document looks very good, looking forward to check the
> implementation and to try it out. I see that hdfs:// is mentioned as the
> backend storage in the documentation, what about s3://. Is it something
> that needed to be considered in the design, or is it handled by RocksDB.
>
> Thanks,
> Vishnu
>
> On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <[hidden email]>
> wrote:
>
> > Hi Vinshnu,
> >
> > We have obtained an initial design of incremental checkpointing [1] and
> > will start working on incremental checkpointing the next week. You can
> > watch the issue  FLINK-5053 [2] to get timely notification of the
> updates.
> > All suggestions are welcome.
> >
> > [1]
> > https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS
> > UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z
> > [2] https://issues.apache.org/jira/browse/FLINK-5053
> >
> > Regards,
> > Xiaogang
> >
> >
> >
> > 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <[hidden email]>:
> >
> > > Vinshnu,
> > > You can find the latest design discussion for incremental checkpoint in
> > > http://apache-flink-mailing-list-archive.1008284.n3.
> > > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-
> > Flink-td15931.html
> > >  @Stefan Richter
> > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter>
> > > and @Xiaogang
> > > Shi
> > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > name=xiaogang.shi>
> > > might
> > > be the right person to provide you its on-going status.
> > >
> > > cheers,
> > > Shaoxuan
> > >
> > > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath <
> > > [hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > Can someone point me to the branch where the ongoing work for
> > incremental
> > > > checkpoint is going on, I would like to try it out even if the work
> is
> > > not
> > > > complete.
> > > >
> > > > I have a use case where the state size increase about ~1gb every 5
> > > minutes.
> > > >
> > > > Thanks,
> > > > Vishnu
> > > >
> > >
> >
>