Hello everyone!
I would like to start a discussion regarding the future of the Database state backend for the streaming API. The main question is whether we want to keep this as a flink-contrib moduel and continue development as part of Flink or should we move it to an outside library. Just as a quick refresher, the DB state backend is designed to store streaming state in JDBC supporting databases (the connector in the flink repo supports MySQL). There is an adapter interface that can be implemented to use different databases other than MySQL. Advantages: - Can store huge states (tested with more than a TB of state with MySQL) - Fast, incremental checkpoints by default - In-memory cache for hot keys - Supports rescaling easily (restarting the job with different parallelism would still work as expected) - Supports database sharding with custom data partitioner - Exposes an easily accessible state view to the outside (will contain uncheckpointed state) Disadvantages: - Lookup/Update might be much slower than local state backend (RocksDB for instance) - Incremental checkpoints are not very well supported with the current runtime so state cleanup/compaction does not happen automatically - Restore speed depends on database indexing as it involves range deletes based on timestamps (this might get slow on very large states) - The caching layer is not memory aware, this should be moved to flink managed memory We developed (and contributed) the DB state backend to be able to handle large state sizes, but have moved to RocksDB for much better read/write performance. I have a pending PR with some improvements to the MySQL adapter but the question is whether we should keep this as part of Flink contrib or move it to an external library. I am personally undecided whether there are strong use cases that would favour this backend over RocksDB. Any opinions? Cheers, Gyula |
On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <[hidden email]> wrote:
> We developed (and contributed) the DB state backend to be able to handle > large state sizes, but have moved to RocksDB for much better read/write > performance. I have a pending PR with some improvements to the MySQL > adapter but the question is whether we should keep this as part of Flink > contrib or move it to an external library. > > I am personally undecided whether there are strong use cases that would > favour this backend over RocksDB. Hey Gyula, thanks for starting this discussion and the great summary of Pros and Cons. I fully agree that it boils down to whether there are strong use cases that would favour the DB backend over RocksDB. Personally, I am in favour of moving the DB state backend out of core Flink: - I think that the DB state backend was a very valuable contribution, but as you've already outlined, RocksDB seems to work better for large out of core state, which was the prime motivation for the DB state backend. - Furthermore, my experience from chatting with you while you were evaluating the DB backend was that there are many parameters to tune for this to work realiably (at least with MySQL). I'm wondering how many users will have the same motivation to actually look into all of these parameters. In my experience, RocksDB provides a better out of the box experience, which is in line with Flink's over all philosophy. It would be possible to just keep the code around in the contrib module, but if we don't have the resources to maintain it (including more documentation and tests), I don't see the point. Since you are the main person driving this, the question is how much time you can/want to invest into this. If you feel like that you personally would really like to keep it in Flink and develop it further here, I would certainly be in favour of keeping it. But at the moment, I don't see this being the case. – Ufuk |
Hi,
if you yourself (Gyula) don’t want to maintain it anymore in the Flink codebase I would vote to move it to an external repository. If you are not using it anymore I’m afraid no one will really work on it. On more thing. When using the DB state backend savepoints don’t work. Cleanup/compaction mess with the stored state and a checkpoint/savepoint of the DB state backend is therefore not a self-contained unit. Cheers, Aljoscha > On 14 Mar 2016, at 12:30, Ufuk Celebi <[hidden email]> wrote: > > On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <[hidden email]> wrote: >> We developed (and contributed) the DB state backend to be able to handle >> large state sizes, but have moved to RocksDB for much better read/write >> performance. I have a pending PR with some improvements to the MySQL >> adapter but the question is whether we should keep this as part of Flink >> contrib or move it to an external library. >> >> I am personally undecided whether there are strong use cases that would >> favour this backend over RocksDB. > > Hey Gyula, > > thanks for starting this discussion and the great summary of Pros and > Cons. I fully agree that it boils down to whether there are strong use > cases that would favour the DB backend over RocksDB. > > Personally, I am in favour of moving the DB state backend out of core Flink: > > - I think that the DB state backend was a very valuable contribution, > but as you've already outlined, RocksDB seems to work better for large > out of core state, which was the prime motivation for the DB state > backend. > > - Furthermore, my experience from chatting with you while you were > evaluating the DB backend was that there are many parameters to tune > for this to work realiably (at least with MySQL). I'm wondering how > many users will have the same motivation to actually look into all of > these parameters. In my experience, RocksDB provides a better out of > the box experience, which is in line with Flink's over all philosophy. > > It would be possible to just keep the code around in the contrib > module, but if we don't have the resources to maintain it (including > more documentation and tests), I don't see the point. > > Since you are the main person driving this, the question is how much > time you can/want to invest into this. If you feel like that you > personally would really like to keep it in Flink and develop it > further here, I would certainly be in favour of keeping it. But at the > moment, I don't see this being the case. > > – Ufuk |
Hey,
Yes, I agree that it is probably best to remove this backend from Flink and make it an external library instead. I will prepare a JIRA and PR. Gyula Aljoscha Krettek <[hidden email]> ezt írta (időpont: 2016. márc. 16., Sze, 10:35): > Hi, > if you yourself (Gyula) don’t want to maintain it anymore in the Flink > codebase I would vote to move it to an external repository. If you are not > using it anymore I’m afraid no one will really work on it. > > On more thing. When using the DB state backend savepoints don’t work. > Cleanup/compaction mess with the stored state and a checkpoint/savepoint of > the DB state backend is therefore not a self-contained unit. > > Cheers, > Aljoscha > > On 14 Mar 2016, at 12:30, Ufuk Celebi <[hidden email]> wrote: > > > > On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <[hidden email]> wrote: > >> We developed (and contributed) the DB state backend to be able to handle > >> large state sizes, but have moved to RocksDB for much better read/write > >> performance. I have a pending PR with some improvements to the MySQL > >> adapter but the question is whether we should keep this as part of Flink > >> contrib or move it to an external library. > >> > >> I am personally undecided whether there are strong use cases that would > >> favour this backend over RocksDB. > > > > Hey Gyula, > > > > thanks for starting this discussion and the great summary of Pros and > > Cons. I fully agree that it boils down to whether there are strong use > > cases that would favour the DB backend over RocksDB. > > > > Personally, I am in favour of moving the DB state backend out of core > Flink: > > > > - I think that the DB state backend was a very valuable contribution, > > but as you've already outlined, RocksDB seems to work better for large > > out of core state, which was the prime motivation for the DB state > > backend. > > > > - Furthermore, my experience from chatting with you while you were > > evaluating the DB backend was that there are many parameters to tune > > for this to work realiably (at least with MySQL). I'm wondering how > > many users will have the same motivation to actually look into all of > > these parameters. In my experience, RocksDB provides a better out of > > the box experience, which is in line with Flink's over all philosophy. > > > > It would be possible to just keep the code around in the contrib > > module, but if we don't have the resources to maintain it (including > > more documentation and tests), I don't see the point. > > > > Since you are the main person driving this, the question is how much > > time you can/want to invest into this. If you feel like that you > > personally would really like to keep it in Flink and develop it > > further here, I would certainly be in favour of keeping it. But at the > > moment, I don't see this being the case. > > > > – Ufuk > > |
Free forum by Nabble | Edit this page |