Hi,
Can someone point me to the branch where the ongoing work for incremental checkpoint is going on, I would like to try it out even if the work is not complete. I have a use case where the state size increase about ~1gb every 5 minutes. Thanks, Vishnu |
Vinshnu,
You can find the latest design discussion for incremental checkpoint in http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-Flink-td15931.html @Stefan Richter <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter> and @Xiaogang Shi <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=xiaogang.shi> might be the right person to provide you its on-going status. cheers, Shaoxuan On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath < [hidden email]> wrote: > Hi, > > Can someone point me to the branch where the ongoing work for incremental > checkpoint is going on, I would like to try it out even if the work is not > complete. > > I have a use case where the state size increase about ~1gb every 5 minutes. > > Thanks, > Vishnu > |
Hi Vinshnu,
We have obtained an initial design of incremental checkpointing [1] and will start working on incremental checkpointing the next week. You can watch the issue FLINK-5053 [2] to get timely notification of the updates. All suggestions are welcome. [1] https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jSUMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z [2] https://issues.apache.org/jira/browse/FLINK-5053 Regards, Xiaogang 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <[hidden email]>: > Vinshnu, > You can find the latest design discussion for incremental checkpoint in > http://apache-flink-mailing-list-archive.1008284.n3. > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in-Flink-td15931.html > @Stefan Richter > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter> > and @Xiaogang > Shi > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=xiaogang.shi> > might > be the right person to provide you its on-going status. > > cheers, > Shaoxuan > > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath < > [hidden email]> wrote: > > > Hi, > > > > Can someone point me to the branch where the ongoing work for incremental > > checkpoint is going on, I would like to try it out even if the work is > not > > complete. > > > > I have a use case where the state size increase about ~1gb every 5 > minutes. > > > > Thanks, > > Vishnu > > > |
Thanks everyone for the reply.
@Xiaogang design document looks very good, looking forward to check the implementation and to try it out. I see that hdfs:// is mentioned as the backend storage in the documentation, what about s3://. Is it something that needed to be considered in the design, or is it handled by RocksDB. Thanks, Vishnu On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <[hidden email]> wrote: > Hi Vinshnu, > > We have obtained an initial design of incremental checkpointing [1] and > will start working on incremental checkpointing the next week. You can > watch the issue FLINK-5053 [2] to get timely notification of the updates. > All suggestions are welcome. > > [1] > https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS > UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z > [2] https://issues.apache.org/jira/browse/FLINK-5053 > > Regards, > Xiaogang > > > > 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <[hidden email]>: > > > Vinshnu, > > You can find the latest design discussion for incremental checkpoint in > > http://apache-flink-mailing-list-archive.1008284.n3. > > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in- > Flink-td15931.html > > @Stefan Richter > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter> > > and @Xiaogang > > Shi > > <https://issues.apache.org/jira/secure/ViewProfile.jspa? > name=xiaogang.shi> > > might > > be the right person to provide you its on-going status. > > > > cheers, > > Shaoxuan > > > > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath < > > [hidden email]> wrote: > > > > > Hi, > > > > > > Can someone point me to the branch where the ongoing work for > incremental > > > checkpoint is going on, I would like to try it out even if the work is > > not > > > complete. > > > > > > I have a use case where the state size increase about ~1gb every 5 > > minutes. > > > > > > Thanks, > > > Vishnu > > > > > > |
Hi!
The concrete storage location (HDFS / S3) should be independent of the imeplementation. The design should work for both. Stephan On Mon, Mar 6, 2017 at 5:38 PM, Vishnu Viswanath < [hidden email]> wrote: > Thanks everyone for the reply. > > @Xiaogang design document looks very good, looking forward to check the > implementation and to try it out. I see that hdfs:// is mentioned as the > backend storage in the documentation, what about s3://. Is it something > that needed to be considered in the design, or is it handled by RocksDB. > > Thanks, > Vishnu > > On Sat, Mar 4, 2017 at 2:58 AM, SHI Xiaogang <[hidden email]> > wrote: > > > Hi Vinshnu, > > > > We have obtained an initial design of incremental checkpointing [1] and > > will start working on incremental checkpointing the next week. You can > > watch the issue FLINK-5053 [2] to get timely notification of the > updates. > > All suggestions are welcome. > > > > [1] > > https://docs.google.com/document/d/1VvvPp09gGdVb9D2wHx6NX99yQK0jS > > UMWHQPrQ_mn520/edit#heading=h.2gshh42txc4z > > [2] https://issues.apache.org/jira/browse/FLINK-5053 > > > > Regards, > > Xiaogang > > > > > > > > 2017-03-04 15:43 GMT+08:00 Shaoxuan Wang <[hidden email]>: > > > > > Vinshnu, > > > You can find the latest design discussion for incremental checkpoint in > > > http://apache-flink-mailing-list-archive.1008284.n3. > > > nabble.com/DISCUSS-Support-Incremental-Checkpointing-in- > > Flink-td15931.html > > > @Stefan Richter > > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=srichter> > > > and @Xiaogang > > > Shi > > > <https://issues.apache.org/jira/secure/ViewProfile.jspa? > > name=xiaogang.shi> > > > might > > > be the right person to provide you its on-going status. > > > > > > cheers, > > > Shaoxuan > > > > > > On Sat, Mar 4, 2017 at 5:14 AM, Vishnu Viswanath < > > > [hidden email]> wrote: > > > > > > > Hi, > > > > > > > > Can someone point me to the branch where the ongoing work for > > incremental > > > > checkpoint is going on, I would like to try it out even if the work > is > > > not > > > > complete. > > > > > > > > I have a use case where the state size increase about ~1gb every 5 > > > minutes. > > > > > > > > Thanks, > > > > Vishnu > > > > > > > > > > |
Free forum by Nabble | Edit this page |