[FLINK-3035] Redis as State Backend
Locked
[FLINK-3035] Redis as State Backend
 
|
Hi
Can you please expand the last comment: "I think, however, that for other reasons we will probably not be able to implement this well. The problem is that we have to somehow get at the state in redis for checkpointing. And if we use only one Redis instance for all states then this will be problematic." - Aljoscha Krettek Any other update on this issue will help, not clear the status. Best, Ovidiu |
Re: [FLINK-3035] Redis as State Backend
|
|
Hi,
there are two basic ideas for implementing a StateBackend based on Redis: 1. Rely on Redis to keep the state, use nothing else. 2. Use Redis to keep the state and checkpoint to some distributed file system (such as HDFS) when checkpointing The first idea seems unwise because Redis is not a "strongly consistent distributed data store" as Elias pointed out on the issue. The second Idea is problematic because there is no easy way to read all state for a given Flink operator from a running Redis instance to store it in HDFS. That's what I was getting at in my comment. Cheers, Aljoscha On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < [hidden email]> wrote: > Hi > > Can you please expand the last comment: > > "I think, however, that for other reasons we will probably not be able to > implement this well. The problem is that we have to somehow get at the > state in redis for checkpointing. And if we use only one Redis instance for > all states then this will be problematic." - Aljoscha Krettek > > Any other update on this issue will help, not clear the status. > > Best, > Ovidiu > > |
RE: [FLINK-3035] Redis as State Backend
|
|
Hi
I missed your reply, thank you for feedback. Agree with 1, that will be possible only with Ramcloud. It is clear the second point. A short question: if you checkpoint the operator's state in hdfs I assume that on failure you are restarting the operator's tasks on other nodes, is that possible with RocksDB? Best, Ovidiu -----Original Message----- From: Aljoscha Krettek [mailto:[hidden email]] Sent: Monday, October 17, 2016 2:51 PM To: [hidden email] Subject: Re: [FLINK-3035] Redis as State Backend Hi, there are two basic ideas for implementing a StateBackend based on Redis: 1. Rely on Redis to keep the state, use nothing else. 2. Use Redis to keep the state and checkpoint to some distributed file system (such as HDFS) when checkpointing The first idea seems unwise because Redis is not a "strongly consistent distributed data store" as Elias pointed out on the issue. The second Idea is problematic because there is no easy way to read all state for a given Flink operator from a running Redis instance to store it in HDFS. That's what I was getting at in my comment. Cheers, Aljoscha On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < [hidden email]> wrote: > Hi > > Can you please expand the last comment: > > "I think, however, that for other reasons we will probably not be able > to implement this well. The problem is that we have to somehow get at > the state in redis for checkpointing. And if we use only one Redis > instance for all states then this will be problematic." - Aljoscha > Krettek > > Any other update on this issue will help, not clear the status. > > Best, > Ovidiu > > |
Re: [FLINK-3035] Redis as State Backend
|
|
Hi,
regarding RocksDB, yes this is possible because RocksDB is essentially only used as an out-of-core hash table. When checkpointing we write everything from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB instance from the data in HDFS. Cheers, Aljoscha On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < [hidden email]> wrote: > Hi > > I missed your reply, thank you for feedback. > Agree with 1, that will be possible only with Ramcloud. > It is clear the second point. > > A short question: if you checkpoint the operator's state in hdfs I assume > that on failure you are > restarting the operator's tasks on other nodes, is that possible with > RocksDB? > > Best, > Ovidiu > > -----Original Message----- > From: Aljoscha Krettek [mailto:[hidden email]] > Sent: Monday, October 17, 2016 2:51 PM > To: [hidden email] > Subject: Re: [FLINK-3035] Redis as State Backend > > Hi, > there are two basic ideas for implementing a StateBackend based on Redis: > 1. Rely on Redis to keep the state, use nothing else. > 2. Use Redis to keep the state and checkpoint to some distributed file > system (such as HDFS) when checkpointing > > The first idea seems unwise because Redis is not a "strongly consistent > distributed data store" as Elias pointed out on the issue. The second Idea > is problematic because there is no easy way to read all state for a given > Flink operator from a running Redis instance to store it in HDFS. That's > what I was getting at in my comment. > > Cheers, > Aljoscha > > On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < > [hidden email]> wrote: > > > Hi > > > > Can you please expand the last comment: > > > > "I think, however, that for other reasons we will probably not be able > > to implement this well. The problem is that we have to somehow get at > > the state in redis for checkpointing. And if we use only one Redis > > instance for all states then this will be problematic." - Aljoscha > > Krettek > > > > Any other update on this issue will help, not clear the status. > > > > Best, > > Ovidiu > > > > > |
Re: [FLINK-3035] Redis as State Backend
|
|
Thank you!
Best, Ovidiu > On 24 Oct 2016, at 16:11, Aljoscha Krettek <[hidden email]> wrote: > > Hi, > regarding RocksDB, yes this is possible because RocksDB is essentially only > used as an out-of-core hash table. When checkpointing we write everything > from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB > instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > [hidden email]> wrote: > >> Hi >> >> I missed your reply, thank you for feedback. >> Agree with 1, that will be possible only with Ramcloud. >> It is clear the second point. >> >> A short question: if you checkpoint the operator's state in hdfs I assume >> that on failure you are >> restarting the operator's tasks on other nodes, is that possible with >> RocksDB? >> >> Best, >> Ovidiu >> >> -----Original Message----- >> From: Aljoscha Krettek [mailto:[hidden email]] >> Sent: Monday, October 17, 2016 2:51 PM >> To: [hidden email] >> Subject: Re: [FLINK-3035] Redis as State Backend >> >> Hi, >> there are two basic ideas for implementing a StateBackend based on Redis: >> 1. Rely on Redis to keep the state, use nothing else. >> 2. Use Redis to keep the state and checkpoint to some distributed file >> system (such as HDFS) when checkpointing >> >> The first idea seems unwise because Redis is not a "strongly consistent >> distributed data store" as Elias pointed out on the issue. The second Idea >> is problematic because there is no easy way to read all state for a given >> Flink operator from a running Redis instance to store it in HDFS. That's >> what I was getting at in my comment. >> >> Cheers, >> Aljoscha >> >> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < >> [hidden email]> wrote: >> >>> Hi >>> >>> Can you please expand the last comment: >>> >>> "I think, however, that for other reasons we will probably not be able >>> to implement this well. The problem is that we have to somehow get at >>> the state in redis for checkpointing. And if we use only one Redis >>> instance for all states then this will be problematic." - Aljoscha >>> Krettek >>> >>> Any other update on this issue will help, not clear the status. >>> >>> Best, >>> Ovidiu >>> >>> >> |
Re: [FLINK-3035] Redis as State Backend
|
|
FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir-
From: Ovidiu-Cristian MARCU <[hidden email]> To: [hidden email] Sent: Tuesday, October 25, 2016 12:40 AM Subject: Re: [FLINK-3035] Redis as State Backend Thank you! Best, Ovidiu > On 24 Oct 2016, at 16:11, Aljoscha Krettek <[hidden email]> wrote: > > Hi, > regarding RocksDB, yes this is possible because RocksDB is essentially only > used as an out-of-core hash table. When checkpointing we write everything > from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB > instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > [hidden email]> wrote: > >> Hi >> >> I missed your reply, thank you for feedback. >> Agree with 1, that will be possible only with Ramcloud. >> It is clear the second point. >> >> A short question: if you checkpoint the operator's state in hdfs I assume >> that on failure you are >> restarting the operator's tasks on other nodes, is that possible with >> RocksDB? >> >> Best, >> Ovidiu >> >> -----Original Message----- >> From: Aljoscha Krettek [mailto:[hidden email]] >> Sent: Monday, October 17, 2016 2:51 PM >> To: [hidden email] >> Subject: Re: [FLINK-3035] Redis as State Backend >> >> Hi, >> there are two basic ideas for implementing a StateBackend based on Redis: >> 1. Rely on Redis to keep the state, use nothing else. >> 2. Use Redis to keep the state and checkpoint to some distributed file >> system (such as HDFS) when checkpointing >> >> The first idea seems unwise because Redis is not a "strongly consistent >> distributed data store" as Elias pointed out on the issue. The second Idea >> is problematic because there is no easy way to read all state for a given >> Flink operator from a running Redis instance to store it in HDFS. That's >> what I was getting at in my comment. >> >> Cheers, >> Aljoscha >> >> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < >> [hidden email]> wrote: >> >>> Hi >>> >>> Can you please expand the last comment: >>> >>> "I think, however, that for other reasons we will probably not be able >>> to implement this well. The problem is that we have to somehow get at >>> the state in redis for checkpointing. And if we use only one Redis >>> instance for all states then this will be problematic." - Aljoscha >>> Krettek >>> >>> Any other update on this issue will help, not clear the status. >>> >>> Best, >>> Ovidiu >>> >>> >> |
RE: [FLINK-3035] Redis as State Backend
|
|
In reply to this post by Aljoscha Krettek-2
Great, thanks!
Best, Ovidiu -----Original Message----- From: Aljoscha Krettek [mailto:[hidden email]] Sent: Monday, October 24, 2016 3:11 PM To: [hidden email] Subject: Re: [FLINK-3035] Redis as State Backend Hi, regarding RocksDB, yes this is possible because RocksDB is essentially only used as an out-of-core hash table. When checkpointing we write everything from RocksDB to HDFS. When restoring we repopulate an empty local RocksDB instance from the data in HDFS. Cheers, Aljoscha On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < [hidden email]> wrote: > Hi > > I missed your reply, thank you for feedback. > Agree with 1, that will be possible only with Ramcloud. > It is clear the second point. > > A short question: if you checkpoint the operator's state in hdfs I > assume that on failure you are restarting the operator's tasks on > other nodes, is that possible with RocksDB? > > Best, > Ovidiu > > -----Original Message----- > From: Aljoscha Krettek [mailto:[hidden email]] > Sent: Monday, October 17, 2016 2:51 PM > To: [hidden email] > Subject: Re: [FLINK-3035] Redis as State Backend > > Hi, > there are two basic ideas for implementing a StateBackend based on Redis: > 1. Rely on Redis to keep the state, use nothing else. > 2. Use Redis to keep the state and checkpoint to some distributed > file system (such as HDFS) when checkpointing > > The first idea seems unwise because Redis is not a "strongly > consistent distributed data store" as Elias pointed out on the issue. > The second Idea is problematic because there is no easy way to read > all state for a given Flink operator from a running Redis instance to > store it in HDFS. That's what I was getting at in my comment. > > Cheers, > Aljoscha > > On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < > [hidden email]> wrote: > > > Hi > > > > Can you please expand the last comment: > > > > "I think, however, that for other reasons we will probably not be > > able to implement this well. The problem is that we have to somehow > > get at the state in redis for checkpointing. And if we use only one > > Redis instance for all states then this will be problematic." - > > Aljoscha Krettek > > > > Any other update on this issue will help, not clear the status. > > > > Best, > > Ovidiu > > > > > |
RE: [FLINK-3035] Redis as State Backend
|
|
In reply to this post by amir bahmanyari
Thanks, how big was your state (GBs)?
Can you share your benchmark/s? Best, Ovidiu -----Original Message----- From: amir bahmanyari [mailto:[hidden email]] Sent: Tuesday, October 25, 2016 7:24 PM To: [hidden email] Subject: Re: [FLINK-3035] Redis as State Backend FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU <[hidden email]> To: [hidden email] Sent: Tuesday, October 25, 2016 12:40 AM Subject: Re: [FLINK-3035] Redis as State Backend Thank you! Best, Ovidiu > On 24 Oct 2016, at 16:11, Aljoscha Krettek <[hidden email]> wrote: > > Hi, > regarding RocksDB, yes this is possible because RocksDB is essentially > only used as an out-of-core hash table. When checkpointing we write > everything from RocksDB to HDFS. When restoring we repopulate an empty > local RocksDB instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > [hidden email]> wrote: > >> Hi >> >> I missed your reply, thank you for feedback. >> Agree with 1, that will be possible only with Ramcloud. >> It is clear the second point. >> >> A short question: if you checkpoint the operator's state in hdfs I >> assume that on failure you are restarting the operator's tasks on >> other nodes, is that possible with RocksDB? >> >> Best, >> Ovidiu >> >> -----Original Message----- >> From: Aljoscha Krettek [mailto:[hidden email]] >> Sent: Monday, October 17, 2016 2:51 PM >> To: [hidden email] >> Subject: Re: [FLINK-3035] Redis as State Backend >> >> Hi, >> there are two basic ideas for implementing a StateBackend based on Redis: >> 1. Rely on Redis to keep the state, use nothing else. >> 2. Use Redis to keep the state and checkpoint to some distributed >> file system (such as HDFS) when checkpointing >> >> The first idea seems unwise because Redis is not a "strongly >> consistent distributed data store" as Elias pointed out on the issue. >> The second Idea is problematic because there is no easy way to read >> all state for a given Flink operator from a running Redis instance to >> store it in HDFS. That's what I was getting at in my comment. >> >> Cheers, >> Aljoscha >> >> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < >> [hidden email]> wrote: >> >>> Hi >>> >>> Can you please expand the last comment: >>> >>> "I think, however, that for other reasons we will probably not be >>> able to implement this well. The problem is that we have to somehow >>> get at the state in redis for checkpointing. And if we use only one >>> Redis instance for all states then this will be problematic." - >>> Aljoscha Krettek >>> >>> Any other update on this issue will help, not clear the status. >>> >>> Best, >>> Ovidiu >>> >>> >> |
Re: [FLINK-3035] Redis as State Backend
|
|
I didnt measure size, just with vs. w/out Redis made a day & night difference in performance so I replaced it with Java concurrent HM objects.Still in progress as far as benchmarking. Have issues with tuning Flink for very high loads...
You can see some of my communications with Aljoscha...Cheers From: Ovidiu Cristian Marcu <[hidden email]> To: "[hidden email]" <[hidden email]>; amir bahmanyari <[hidden email]> Sent: Monday, November 7, 2016 3:56 AM Subject: RE: [FLINK-3035] Redis as State Backend Thanks, how big was your state (GBs)? Can you share your benchmark/s? Best, Ovidiu -----Original Message----- From: amir bahmanyari [mailto:[hidden email]] Sent: Tuesday, October 25, 2016 7:24 PM To: [hidden email] Subject: Re: [FLINK-3035] Redis as State Backend FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU <[hidden email]> To: [hidden email] Sent: Tuesday, October 25, 2016 12:40 AM Subject: Re: [FLINK-3035] Redis as State Backend Thank you! Best, Ovidiu > On 24 Oct 2016, at 16:11, Aljoscha Krettek <[hidden email]> wrote: > > Hi, > regarding RocksDB, yes this is possible because RocksDB is essentially > only used as an out-of-core hash table. When checkpointing we write > everything from RocksDB to HDFS. When restoring we repopulate an empty > local RocksDB instance from the data in HDFS. > > Cheers, > Aljoscha > > On Fri, 21 Oct 2016 at 11:24 Ovidiu Cristian Marcu < > [hidden email]> wrote: > >> Hi >> >> I missed your reply, thank you for feedback. >> Agree with 1, that will be possible only with Ramcloud. >> It is clear the second point. >> >> A short question: if you checkpoint the operator's state in hdfs I >> assume that on failure you are restarting the operator's tasks on >> other nodes, is that possible with RocksDB? >> >> Best, >> Ovidiu >> >> -----Original Message----- >> From: Aljoscha Krettek [mailto:[hidden email]] >> Sent: Monday, October 17, 2016 2:51 PM >> To: [hidden email] >> Subject: Re: [FLINK-3035] Redis as State Backend >> >> Hi, >> there are two basic ideas for implementing a StateBackend based on Redis: >> 1. Rely on Redis to keep the state, use nothing else. >> 2. Use Redis to keep the state and checkpoint to some distributed >> file system (such as HDFS) when checkpointing >> >> The first idea seems unwise because Redis is not a "strongly >> consistent distributed data store" as Elias pointed out on the issue. >> The second Idea is problematic because there is no easy way to read >> all state for a given Flink operator from a running Redis instance to >> store it in HDFS. That's what I was getting at in my comment. >> >> Cheers, >> Aljoscha >> >> On Fri, 7 Oct 2016 at 17:19 Ovidiu Cristian Marcu < >> [hidden email]> wrote: >> >>> Hi >>> >>> Can you please expand the last comment: >>> >>> "I think, however, that for other reasons we will probably not be >>> able to implement this well. The problem is that we have to somehow >>> get at the state in redis for checkpointing. And if we use only one >>> Redis instance for all states then this will be problematic." - >>> Aljoscha Krettek >>> >>> Any other update on this issue will help, not clear the status. >>> >>> Best, >>> Ovidiu >>> >>> >> |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |