Hi All!
I want to share with you a little project we have been working on at King (with some help from some dataArtisans folks). I think this would be a valuable addition to Flink and solve a bunch of outstanding production use-cases and headaches around state bootstrapping and state analytics. We have built a quick and dirty POC implementation on top of Flink 1.6, please check the README for some nice examples to get a quick idea: https://github.com/king/bravo *Short story* Bravo is a convenient state reader and writer library leveraging the Flink’s batch processing capabilities. It supports processing and writing Flink streaming savepoints. At the moment it only supports processing RocksDB savepoints but this can be extended in the future for other state backends and checkpoint types. Our goal is to cover a few basic features: - Converting keyed states to Flink DataSets for processing and analytics - Reading/Writing non-keyed operators states - Bootstrap keyed states from Flink DataSets and create new valid savepoints - Transform existing savepoints by replacing/changing some states Some example use-cases: - Point-in-time state analytics across all operators and keys - Bootstrap state of a streaming job from external resources such as reading from database/filesystem - Validate and potentially repair corrupted state of a streaming job - Change max parallelism of a job Our main goal is to start working together with other Flink production users and make this something useful that can be part of Flink. So if you have use-cases please talk to us :) I have also started a google doc which contains a little bit more info than the readme and could be a starting place for discussions: https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing I know there are a bunch of rough edges and bugs (and no tests) but our motto is: If you are not embarrassed, you released too late :) Please let me know what you think! Cheers, Gyula |
Hi,
Very huge +1 from my side. I found lack of such tool/possibility as a big problem for long term maintainability of Flink jobs. In the long run, I would be delight to see Flink SQL support for those things as well. Ad hoc analysis is one of the prime use case of SQL. This tool would make analysis possible, while SQL could make them easy to use and shorten the feedback loop. Especially in cases when you are not sure what you are looking for in the state. Just to clarify. Is your end goal to contribute such tool to apache Flink or do you want it to be separate tool? Piotrek > On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > > Hi All! > > I want to share with you a little project we have been working on at King > (with some help from some dataArtisans folks). I think this would be a > valuable addition to Flink and solve a bunch of outstanding production > use-cases and headaches around state bootstrapping and state analytics. > > We have built a quick and dirty POC implementation on top of Flink 1.6, > please check the README for some nice examples to get a quick idea: > > https://github.com/king/bravo > > *Short story* > Bravo is a convenient state reader and writer library leveraging the > Flink’s batch processing capabilities. It supports processing and writing > Flink streaming savepoints. At the moment it only supports processing > RocksDB savepoints but this can be extended in the future for other state > backends and checkpoint types. > > Our goal is to cover a few basic features: > > - Converting keyed states to Flink DataSets for processing and analytics > - Reading/Writing non-keyed operators states > - Bootstrap keyed states from Flink DataSets and create new valid > savepoints > - Transform existing savepoints by replacing/changing some states > > > Some example use-cases: > > - Point-in-time state analytics across all operators and keys > - Bootstrap state of a streaming job from external resources such as > reading from database/filesystem > - Validate and potentially repair corrupted state of a streaming job > - Change max parallelism of a job > > > Our main goal is to start working together with other Flink production > users and make this something useful that can be part of Flink. So if you > have use-cases please talk to us :) > I have also started a google doc which contains a little bit more info than > the readme and could be a starting place for discussions: > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > I know there are a bunch of rough edges and bugs (and no tests) but our > motto is: If you are not embarrassed, you released too late :) > > Please let me know what you think! > > Cheers, > Gyula |
Thanks for the feedback :) I agree that combining this with SQL would give
an extremely nice layer to analyse the states. Our goal is to contribute this to Flink, I think this should live as part of the Flink project to make deeper intergration possible in the long run. Of course a pre-requisite for this is that there is enough production interest for such a tool, but I believe there should be :) Gyula Piotr Nowojski <[hidden email]> ezt írta (időpont: 2018. aug. 17., P, 15:07): > Hi, > > Very huge +1 from my side. I found lack of such tool/possibility as a big > problem for long term maintainability of Flink jobs. > > In the long run, I would be delight to see Flink SQL support for those > things as well. Ad hoc analysis is one of the prime use case of SQL. This > tool would make analysis possible, while SQL could make them easy to use > and shorten the feedback loop. Especially in cases when you are not sure > what you are looking for in the state. > > Just to clarify. Is your end goal to contribute such tool to apache Flink > or do you want it to be separate tool? > > Piotrek > > > On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > > > > Hi All! > > > > I want to share with you a little project we have been working on at King > > (with some help from some dataArtisans folks). I think this would be a > > valuable addition to Flink and solve a bunch of outstanding production > > use-cases and headaches around state bootstrapping and state analytics. > > > > We have built a quick and dirty POC implementation on top of Flink 1.6, > > please check the README for some nice examples to get a quick idea: > > > > https://github.com/king/bravo > > > > *Short story* > > Bravo is a convenient state reader and writer library leveraging the > > Flink’s batch processing capabilities. It supports processing and writing > > Flink streaming savepoints. At the moment it only supports processing > > RocksDB savepoints but this can be extended in the future for other state > > backends and checkpoint types. > > > > Our goal is to cover a few basic features: > > > > - Converting keyed states to Flink DataSets for processing and > analytics > > - Reading/Writing non-keyed operators states > > - Bootstrap keyed states from Flink DataSets and create new valid > > savepoints > > - Transform existing savepoints by replacing/changing some states > > > > > > Some example use-cases: > > > > - Point-in-time state analytics across all operators and keys > > - Bootstrap state of a streaming job from external resources such as > > reading from database/filesystem > > - Validate and potentially repair corrupted state of a streaming job > > - Change max parallelism of a job > > > > > > Our main goal is to start working together with other Flink production > > users and make this something useful that can be part of Flink. So if you > > have use-cases please talk to us :) > > I have also started a google doc which contains a little bit more info > than > > the readme and could be a starting place for discussions: > > > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > > > I know there are a bunch of rough edges and bugs (and no tests) but our > > motto is: If you are not embarrassed, you released too late :) > > > > Please let me know what you think! > > > > Cheers, > > Gyula > > |
In reply to this post by Gyula Fóra
This is great, Gyula! A colleague here at Lyft has also done some work
around bootstrapping DataStream programs and we've also talked a bit about doing this by running DataSet programs. On Fri, Aug 17, 2018 at 3:28 AM, Gyula Fóra <[hidden email]> wrote: > Hi All! > > I want to share with you a little project we have been working on at King > (with some help from some dataArtisans folks). I think this would be a > valuable addition to Flink and solve a bunch of outstanding production > use-cases and headaches around state bootstrapping and state analytics. > > We have built a quick and dirty POC implementation on top of Flink 1.6, > please check the README for some nice examples to get a quick idea: > > https://github.com/king/bravo > > *Short story* > Bravo is a convenient state reader and writer library leveraging the > Flink’s batch processing capabilities. It supports processing and writing > Flink streaming savepoints. At the moment it only supports processing > RocksDB savepoints but this can be extended in the future for other state > backends and checkpoint types. > > Our goal is to cover a few basic features: > > - Converting keyed states to Flink DataSets for processing and analytics > - Reading/Writing non-keyed operators states > - Bootstrap keyed states from Flink DataSets and create new valid > savepoints > - Transform existing savepoints by replacing/changing some states > > > Some example use-cases: > > - Point-in-time state analytics across all operators and keys > - Bootstrap state of a streaming job from external resources such as > reading from database/filesystem > - Validate and potentially repair corrupted state of a streaming job > - Change max parallelism of a job > > > Our main goal is to start working together with other Flink production > users and make this something useful that can be part of Flink. So if you > have use-cases please talk to us :) > I have also started a google doc which contains a little bit more info than > the readme and could be a starting place for discussions: > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpw > dhqBMr-ppkFL5E/edit?usp=sharing > > I know there are a bunch of rough edges and bugs (and no tests) but our > motto is: If you are not embarrassed, you released too late :) > > Please let me know what you think! > > Cheers, > Gyula > |
In reply to this post by Gyula Fóra
+1
Might also be a good start to implement queryable stream state with snapshot isolation using that mechanism. Paris > On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > > Hi All! > > I want to share with you a little project we have been working on at King > (with some help from some dataArtisans folks). I think this would be a > valuable addition to Flink and solve a bunch of outstanding production > use-cases and headaches around state bootstrapping and state analytics. > > We have built a quick and dirty POC implementation on top of Flink 1.6, > please check the README for some nice examples to get a quick idea: > > https://github.com/king/bravo > > *Short story* > Bravo is a convenient state reader and writer library leveraging the > Flink’s batch processing capabilities. It supports processing and writing > Flink streaming savepoints. At the moment it only supports processing > RocksDB savepoints but this can be extended in the future for other state > backends and checkpoint types. > > Our goal is to cover a few basic features: > > - Converting keyed states to Flink DataSets for processing and analytics > - Reading/Writing non-keyed operators states > - Bootstrap keyed states from Flink DataSets and create new valid > savepoints > - Transform existing savepoints by replacing/changing some states > > > Some example use-cases: > > - Point-in-time state analytics across all operators and keys > - Bootstrap state of a streaming job from external resources such as > reading from database/filesystem > - Validate and potentially repair corrupted state of a streaming job > - Change max parallelism of a job > > > Our main goal is to start working together with other Flink production > users and make this something useful that can be part of Flink. So if you > have use-cases please talk to us :) > I have also started a google doc which contains a little bit more info than > the readme and could be a starting place for discussions: > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > I know there are a bunch of rough edges and bugs (and no tests) but our > motto is: If you are not embarrassed, you released too late :) > > Please let me know what you think! > > Cheers, > Gyula |
Hi all!
Thanks for the feedback and I'm happy there is some interest :) Tomorrow I will start improving the proposal based on the feedback and will get back to work. If you are interested working together in this please ping me and we can discuss some ideas/plans and how to share work. Cheers, Gyula Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., Szo, 9:03): > +1 > > Might also be a good start to implement queryable stream state with > snapshot isolation using that mechanism. > > Paris > > > On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > > > > Hi All! > > > > I want to share with you a little project we have been working on at King > > (with some help from some dataArtisans folks). I think this would be a > > valuable addition to Flink and solve a bunch of outstanding production > > use-cases and headaches around state bootstrapping and state analytics. > > > > We have built a quick and dirty POC implementation on top of Flink 1.6, > > please check the README for some nice examples to get a quick idea: > > > > https://github.com/king/bravo > > > > *Short story* > > Bravo is a convenient state reader and writer library leveraging the > > Flink’s batch processing capabilities. It supports processing and writing > > Flink streaming savepoints. At the moment it only supports processing > > RocksDB savepoints but this can be extended in the future for other state > > backends and checkpoint types. > > > > Our goal is to cover a few basic features: > > > > - Converting keyed states to Flink DataSets for processing and > analytics > > - Reading/Writing non-keyed operators states > > - Bootstrap keyed states from Flink DataSets and create new valid > > savepoints > > - Transform existing savepoints by replacing/changing some states > > > > > > Some example use-cases: > > > > - Point-in-time state analytics across all operators and keys > > - Bootstrap state of a streaming job from external resources such as > > reading from database/filesystem > > - Validate and potentially repair corrupted state of a streaming job > > - Change max parallelism of a job > > > > > > Our main goal is to start working together with other Flink production > > users and make this something useful that can be part of Flink. So if you > > have use-cases please talk to us :) > > I have also started a google doc which contains a little bit more info > than > > the readme and could be a starting place for discussions: > > > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > > > I know there are a bunch of rough edges and bugs (and no tests) but our > > motto is: If you are not embarrassed, you released too late :) > > > > Please let me know what you think! > > > > Cheers, > > Gyula > > |
+1 I'd like to have something like this in Flink a lot!
> On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> wrote: > > Hi all! > > Thanks for the feedback and I'm happy there is some interest :) > Tomorrow I will start improving the proposal based on the feedback and will > get back to work. > > If you are interested working together in this please ping me and we can > discuss some ideas/plans and how to share work. > > Cheers, > Gyula > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., Szo, 9:03): > >> +1 >> >> Might also be a good start to implement queryable stream state with >> snapshot isolation using that mechanism. >> >> Paris >> >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: >>> >>> Hi All! >>> >>> I want to share with you a little project we have been working on at King >>> (with some help from some dataArtisans folks). I think this would be a >>> valuable addition to Flink and solve a bunch of outstanding production >>> use-cases and headaches around state bootstrapping and state analytics. >>> >>> We have built a quick and dirty POC implementation on top of Flink 1.6, >>> please check the README for some nice examples to get a quick idea: >>> >>> https://github.com/king/bravo >>> >>> *Short story* >>> Bravo is a convenient state reader and writer library leveraging the >>> Flink’s batch processing capabilities. It supports processing and writing >>> Flink streaming savepoints. At the moment it only supports processing >>> RocksDB savepoints but this can be extended in the future for other state >>> backends and checkpoint types. >>> >>> Our goal is to cover a few basic features: >>> >>> - Converting keyed states to Flink DataSets for processing and >> analytics >>> - Reading/Writing non-keyed operators states >>> - Bootstrap keyed states from Flink DataSets and create new valid >>> savepoints >>> - Transform existing savepoints by replacing/changing some states >>> >>> >>> Some example use-cases: >>> >>> - Point-in-time state analytics across all operators and keys >>> - Bootstrap state of a streaming job from external resources such as >>> reading from database/filesystem >>> - Validate and potentially repair corrupted state of a streaming job >>> - Change max parallelism of a job >>> >>> >>> Our main goal is to start working together with other Flink production >>> users and make this something useful that can be part of Flink. So if you >>> have use-cases please talk to us :) >>> I have also started a google doc which contains a little bit more info >> than >>> the readme and could be a starting place for discussions: >>> >>> >> https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing >>> >>> I know there are a bunch of rough edges and bugs (and no tests) but our >>> motto is: If you are not embarrassed, you released too late :) >>> >>> Please let me know what you think! >>> >>> Cheers, >>> Gyula >> >> |
big +1 for this feature. A tool to get your state out of and into Flink
will be tremendously helpful. On Mon, Aug 20, 2018 at 10:21 AM Aljoscha Krettek <[hidden email]> wrote: > +1 I'd like to have something like this in Flink a lot! > > > On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> wrote: > > > > Hi all! > > > > Thanks for the feedback and I'm happy there is some interest :) > > Tomorrow I will start improving the proposal based on the feedback and > will > > get back to work. > > > > If you are interested working together in this please ping me and we can > > discuss some ideas/plans and how to share work. > > > > Cheers, > > Gyula > > > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., Szo, > 9:03): > > > >> +1 > >> > >> Might also be a good start to implement queryable stream state with > >> snapshot isolation using that mechanism. > >> > >> Paris > >> > >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > >>> > >>> Hi All! > >>> > >>> I want to share with you a little project we have been working on at > King > >>> (with some help from some dataArtisans folks). I think this would be a > >>> valuable addition to Flink and solve a bunch of outstanding production > >>> use-cases and headaches around state bootstrapping and state analytics. > >>> > >>> We have built a quick and dirty POC implementation on top of Flink 1.6, > >>> please check the README for some nice examples to get a quick idea: > >>> > >>> https://github.com/king/bravo > >>> > >>> *Short story* > >>> Bravo is a convenient state reader and writer library leveraging the > >>> Flink’s batch processing capabilities. It supports processing and > writing > >>> Flink streaming savepoints. At the moment it only supports processing > >>> RocksDB savepoints but this can be extended in the future for other > state > >>> backends and checkpoint types. > >>> > >>> Our goal is to cover a few basic features: > >>> > >>> - Converting keyed states to Flink DataSets for processing and > >> analytics > >>> - Reading/Writing non-keyed operators states > >>> - Bootstrap keyed states from Flink DataSets and create new valid > >>> savepoints > >>> - Transform existing savepoints by replacing/changing some states > >>> > >>> > >>> Some example use-cases: > >>> > >>> - Point-in-time state analytics across all operators and keys > >>> - Bootstrap state of a streaming job from external resources such as > >>> reading from database/filesystem > >>> - Validate and potentially repair corrupted state of a streaming job > >>> - Change max parallelism of a job > >>> > >>> > >>> Our main goal is to start working together with other Flink production > >>> users and make this something useful that can be part of Flink. So if > you > >>> have use-cases please talk to us :) > >>> I have also started a google doc which contains a little bit more info > >> than > >>> the readme and could be a starting place for discussions: > >>> > >>> > >> > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > >>> > >>> I know there are a bunch of rough edges and bugs (and no tests) but our > >>> motto is: If you are not embarrassed, you released too late :) > >>> > >>> Please let me know what you think! > >>> > >>> Cheers, > >>> Gyula > >> > >> > > |
+1. Being able to analyze the state is a huge operational advantage.
Thanks Gyula for the POC and I would be very interested in contributing to the work. -- Rong On Tue, Aug 21, 2018 at 4:26 AM Till Rohrmann <[hidden email]> wrote: > big +1 for this feature. A tool to get your state out of and into Flink > will be tremendously helpful. > > On Mon, Aug 20, 2018 at 10:21 AM Aljoscha Krettek <[hidden email]> > wrote: > > > +1 I'd like to have something like this in Flink a lot! > > > > > On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> wrote: > > > > > > Hi all! > > > > > > Thanks for the feedback and I'm happy there is some interest :) > > > Tomorrow I will start improving the proposal based on the feedback and > > will > > > get back to work. > > > > > > If you are interested working together in this please ping me and we > can > > > discuss some ideas/plans and how to share work. > > > > > > Cheers, > > > Gyula > > > > > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., Szo, > > 9:03): > > > > > >> +1 > > >> > > >> Might also be a good start to implement queryable stream state with > > >> snapshot isolation using that mechanism. > > >> > > >> Paris > > >> > > >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > > >>> > > >>> Hi All! > > >>> > > >>> I want to share with you a little project we have been working on at > > King > > >>> (with some help from some dataArtisans folks). I think this would be > a > > >>> valuable addition to Flink and solve a bunch of outstanding > production > > >>> use-cases and headaches around state bootstrapping and state > analytics. > > >>> > > >>> We have built a quick and dirty POC implementation on top of Flink > 1.6, > > >>> please check the README for some nice examples to get a quick idea: > > >>> > > >>> https://github.com/king/bravo > > >>> > > >>> *Short story* > > >>> Bravo is a convenient state reader and writer library leveraging the > > >>> Flink’s batch processing capabilities. It supports processing and > > writing > > >>> Flink streaming savepoints. At the moment it only supports processing > > >>> RocksDB savepoints but this can be extended in the future for other > > state > > >>> backends and checkpoint types. > > >>> > > >>> Our goal is to cover a few basic features: > > >>> > > >>> - Converting keyed states to Flink DataSets for processing and > > >> analytics > > >>> - Reading/Writing non-keyed operators states > > >>> - Bootstrap keyed states from Flink DataSets and create new valid > > >>> savepoints > > >>> - Transform existing savepoints by replacing/changing some states > > >>> > > >>> > > >>> Some example use-cases: > > >>> > > >>> - Point-in-time state analytics across all operators and keys > > >>> - Bootstrap state of a streaming job from external resources such as > > >>> reading from database/filesystem > > >>> - Validate and potentially repair corrupted state of a streaming job > > >>> - Change max parallelism of a job > > >>> > > >>> > > >>> Our main goal is to start working together with other Flink > production > > >>> users and make this something useful that can be part of Flink. So if > > you > > >>> have use-cases please talk to us :) > > >>> I have also started a google doc which contains a little bit more > info > > >> than > > >>> the readme and could be a starting place for discussions: > > >>> > > >>> > > >> > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > >>> > > >>> I know there are a bunch of rough edges and bugs (and no tests) but > our > > >>> motto is: If you are not embarrassed, you released too late :) > > >>> > > >>> Please let me know what you think! > > >>> > > >>> Cheers, > > >>> Gyula > > >> > > >> > > > > > |
Thanks,
I guess the first thing that would be great help from anyone interested in helping is to try it for some streaming state :) We have tested these tools at King to analyze, transform and perform some aggregations on our user-states. The major limitation is that it requires RocksDB savepoints to work but other than that we successfully analyzed a few hundred gigabytes of state including reading keyed, and broadcast states from different operators. Also you need to have a savepoint before you can create a new savepoint (with whatever state). Once we have some people who have played with it we can probably greatly improve the API and user experience as it is pretty low level at the moment. I suggest we use the King git repo <https://github.com/king/bravo> for now to track some features before it is in a shape that deserves a Flink PR. We are super happy to take any improvements, code contributions from anyone so dont hesitate to reach out to me if you have some ideas. Gyula Rong Rong <[hidden email]> ezt írta (időpont: 2018. aug. 22., Sze, 17:06): > +1. Being able to analyze the state is a huge operational advantage. > Thanks Gyula for the POC and I would be very interested in contributing to > the work. > > -- > Rong > > On Tue, Aug 21, 2018 at 4:26 AM Till Rohrmann <[hidden email]> > wrote: > > > big +1 for this feature. A tool to get your state out of and into Flink > > will be tremendously helpful. > > > > On Mon, Aug 20, 2018 at 10:21 AM Aljoscha Krettek <[hidden email]> > > wrote: > > > > > +1 I'd like to have something like this in Flink a lot! > > > > > > > On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> wrote: > > > > > > > > Hi all! > > > > > > > > Thanks for the feedback and I'm happy there is some interest :) > > > > Tomorrow I will start improving the proposal based on the feedback > and > > > will > > > > get back to work. > > > > > > > > If you are interested working together in this please ping me and we > > can > > > > discuss some ideas/plans and how to share work. > > > > > > > > Cheers, > > > > Gyula > > > > > > > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., > Szo, > > > 9:03): > > > > > > > >> +1 > > > >> > > > >> Might also be a good start to implement queryable stream state with > > > >> snapshot isolation using that mechanism. > > > >> > > > >> Paris > > > >> > > > >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> wrote: > > > >>> > > > >>> Hi All! > > > >>> > > > >>> I want to share with you a little project we have been working on > at > > > King > > > >>> (with some help from some dataArtisans folks). I think this would > be > > a > > > >>> valuable addition to Flink and solve a bunch of outstanding > > production > > > >>> use-cases and headaches around state bootstrapping and state > > analytics. > > > >>> > > > >>> We have built a quick and dirty POC implementation on top of Flink > > 1.6, > > > >>> please check the README for some nice examples to get a quick idea: > > > >>> > > > >>> https://github.com/king/bravo > > > >>> > > > >>> *Short story* > > > >>> Bravo is a convenient state reader and writer library leveraging > the > > > >>> Flink’s batch processing capabilities. It supports processing and > > > writing > > > >>> Flink streaming savepoints. At the moment it only supports > processing > > > >>> RocksDB savepoints but this can be extended in the future for other > > > state > > > >>> backends and checkpoint types. > > > >>> > > > >>> Our goal is to cover a few basic features: > > > >>> > > > >>> - Converting keyed states to Flink DataSets for processing and > > > >> analytics > > > >>> - Reading/Writing non-keyed operators states > > > >>> - Bootstrap keyed states from Flink DataSets and create new valid > > > >>> savepoints > > > >>> - Transform existing savepoints by replacing/changing some states > > > >>> > > > >>> > > > >>> Some example use-cases: > > > >>> > > > >>> - Point-in-time state analytics across all operators and keys > > > >>> - Bootstrap state of a streaming job from external resources such > as > > > >>> reading from database/filesystem > > > >>> - Validate and potentially repair corrupted state of a streaming > job > > > >>> - Change max parallelism of a job > > > >>> > > > >>> > > > >>> Our main goal is to start working together with other Flink > > production > > > >>> users and make this something useful that can be part of Flink. So > if > > > you > > > >>> have use-cases please talk to us :) > > > >>> I have also started a google doc which contains a little bit more > > info > > > >> than > > > >>> the readme and could be a starting place for discussions: > > > >>> > > > >>> > > > >> > > > > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > > >>> > > > >>> I know there are a bunch of rough edges and bugs (and no tests) but > > our > > > >>> motto is: If you are not embarrassed, you released too late :) > > > >>> > > > >>> Please let me know what you think! > > > >>> > > > >>> Cheers, > > > >>> Gyula > > > >> > > > >> > > > > > > > > > |
+1 on the tooling. Also, you mentioned about state bootstrapping problem.
Could you please elaborate on how we can leverage the tooling to solve state bootstrapping? I think this is a common problem to stream processing, and it will be great the community can work on it. Thanks. Shuyi On Wed, Aug 22, 2018 at 11:51 AM Gyula Fóra <[hidden email]> wrote: > Thanks, > > I guess the first thing that would be great help from anyone interested in > helping is to try it for some streaming state :) > > We have tested these tools at King to analyze, transform and perform some > aggregations on our user-states. The major limitation is that it requires > RocksDB savepoints to work but other than that we successfully analyzed a > few hundred gigabytes of state including reading keyed, and broadcast > states from different operators. Also you need to have a savepoint before > you can create a new savepoint (with whatever state). > > Once we have some people who have played with it we can probably greatly > improve the API and user experience as it is pretty low level at the > moment. I suggest we use the King git repo <https://github.com/king/bravo> > for > now to track some features before it is in a shape that deserves a Flink > PR. We are super happy to take any improvements, code contributions from > anyone so dont hesitate to reach out to me if you have some ideas. > > Gyula > > > Rong Rong <[hidden email]> ezt írta (időpont: 2018. aug. 22., Sze, > 17:06): > > > +1. Being able to analyze the state is a huge operational advantage. > > Thanks Gyula for the POC and I would be very interested in contributing > to > > the work. > > > > -- > > Rong > > > > On Tue, Aug 21, 2018 at 4:26 AM Till Rohrmann <[hidden email]> > > wrote: > > > > > big +1 for this feature. A tool to get your state out of and into Flink > > > will be tremendously helpful. > > > > > > On Mon, Aug 20, 2018 at 10:21 AM Aljoscha Krettek <[hidden email] > > > > > wrote: > > > > > > > +1 I'd like to have something like this in Flink a lot! > > > > > > > > > On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> > wrote: > > > > > > > > > > Hi all! > > > > > > > > > > Thanks for the feedback and I'm happy there is some interest :) > > > > > Tomorrow I will start improving the proposal based on the feedback > > and > > > > will > > > > > get back to work. > > > > > > > > > > If you are interested working together in this please ping me and > we > > > can > > > > > discuss some ideas/plans and how to share work. > > > > > > > > > > Cheers, > > > > > Gyula > > > > > > > > > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., > > Szo, > > > > 9:03): > > > > > > > > > >> +1 > > > > >> > > > > >> Might also be a good start to implement queryable stream state > with > > > > >> snapshot isolation using that mechanism. > > > > >> > > > > >> Paris > > > > >> > > > > >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> > wrote: > > > > >>> > > > > >>> Hi All! > > > > >>> > > > > >>> I want to share with you a little project we have been working on > > at > > > > King > > > > >>> (with some help from some dataArtisans folks). I think this would > > be > > > a > > > > >>> valuable addition to Flink and solve a bunch of outstanding > > > production > > > > >>> use-cases and headaches around state bootstrapping and state > > > analytics. > > > > >>> > > > > >>> We have built a quick and dirty POC implementation on top of > Flink > > > 1.6, > > > > >>> please check the README for some nice examples to get a quick > idea: > > > > >>> > > > > >>> https://github.com/king/bravo > > > > >>> > > > > >>> *Short story* > > > > >>> Bravo is a convenient state reader and writer library leveraging > > the > > > > >>> Flink’s batch processing capabilities. It supports processing and > > > > writing > > > > >>> Flink streaming savepoints. At the moment it only supports > > processing > > > > >>> RocksDB savepoints but this can be extended in the future for > other > > > > state > > > > >>> backends and checkpoint types. > > > > >>> > > > > >>> Our goal is to cover a few basic features: > > > > >>> > > > > >>> - Converting keyed states to Flink DataSets for processing and > > > > >> analytics > > > > >>> - Reading/Writing non-keyed operators states > > > > >>> - Bootstrap keyed states from Flink DataSets and create new > valid > > > > >>> savepoints > > > > >>> - Transform existing savepoints by replacing/changing some > states > > > > >>> > > > > >>> > > > > >>> Some example use-cases: > > > > >>> > > > > >>> - Point-in-time state analytics across all operators and keys > > > > >>> - Bootstrap state of a streaming job from external resources > such > > as > > > > >>> reading from database/filesystem > > > > >>> - Validate and potentially repair corrupted state of a streaming > > job > > > > >>> - Change max parallelism of a job > > > > >>> > > > > >>> > > > > >>> Our main goal is to start working together with other Flink > > > production > > > > >>> users and make this something useful that can be part of Flink. > So > > if > > > > you > > > > >>> have use-cases please talk to us :) > > > > >>> I have also started a google doc which contains a little bit more > > > info > > > > >> than > > > > >>> the readme and could be a starting place for discussions: > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > > > >>> > > > > >>> I know there are a bunch of rough edges and bugs (and no tests) > but > > > our > > > > >>> motto is: If you are not embarrassed, you released too late :) > > > > >>> > > > > >>> Please let me know what you think! > > > > >>> > > > > >>> Cheers, > > > > >>> Gyula > > > > >> > > > > >> > > > > > > > > > > > > > > -- "So you have to trust that the dots will somehow connect in your future." |
Hi Shuyi,
The tool allows you to convert a Flink DataSet containing the individual state rows, (key, state) pairs, into a state for a streaming operator. So if you want to bootstrap your state from data on HDFS, you would read the file in Flink, convert it to the required DataSet format, then you can write the state for the desired operator. As you can read many different input formats in Flink this gives large flexibility to where you can bootstrap the state from. Gyula Shuyi Chen <[hidden email]> ezt írta (időpont: 2018. aug. 22., Sze, 21:28): > +1 on the tooling. Also, you mentioned about state bootstrapping problem. > Could you please elaborate on how we can leverage the tooling to solve > state bootstrapping? I think this is a common problem to stream processing, > and it will be great the community can work on it. Thanks. > > Shuyi > > On Wed, Aug 22, 2018 at 11:51 AM Gyula Fóra <[hidden email]> wrote: > > > Thanks, > > > > I guess the first thing that would be great help from anyone interested > in > > helping is to try it for some streaming state :) > > > > We have tested these tools at King to analyze, transform and perform some > > aggregations on our user-states. The major limitation is that it requires > > RocksDB savepoints to work but other than that we successfully analyzed a > > few hundred gigabytes of state including reading keyed, and broadcast > > states from different operators. Also you need to have a savepoint before > > you can create a new savepoint (with whatever state). > > > > Once we have some people who have played with it we can probably greatly > > improve the API and user experience as it is pretty low level at the > > moment. I suggest we use the King git repo < > https://github.com/king/bravo> > > for > > now to track some features before it is in a shape that deserves a Flink > > PR. We are super happy to take any improvements, code contributions from > > anyone so dont hesitate to reach out to me if you have some ideas. > > > > Gyula > > > > > > Rong Rong <[hidden email]> ezt írta (időpont: 2018. aug. 22., Sze, > > 17:06): > > > > > +1. Being able to analyze the state is a huge operational advantage. > > > Thanks Gyula for the POC and I would be very interested in contributing > > to > > > the work. > > > > > > -- > > > Rong > > > > > > On Tue, Aug 21, 2018 at 4:26 AM Till Rohrmann <[hidden email]> > > > wrote: > > > > > > > big +1 for this feature. A tool to get your state out of and into > Flink > > > > will be tremendously helpful. > > > > > > > > On Mon, Aug 20, 2018 at 10:21 AM Aljoscha Krettek < > [hidden email] > > > > > > > wrote: > > > > > > > > > +1 I'd like to have something like this in Flink a lot! > > > > > > > > > > > On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> > > wrote: > > > > > > > > > > > > Hi all! > > > > > > > > > > > > Thanks for the feedback and I'm happy there is some interest :) > > > > > > Tomorrow I will start improving the proposal based on the > feedback > > > and > > > > > will > > > > > > get back to work. > > > > > > > > > > > > If you are interested working together in this please ping me and > > we > > > > can > > > > > > discuss some ideas/plans and how to share work. > > > > > > > > > > > > Cheers, > > > > > > Gyula > > > > > > > > > > > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. 18., > > > Szo, > > > > > 9:03): > > > > > > > > > > > >> +1 > > > > > >> > > > > > >> Might also be a good start to implement queryable stream state > > with > > > > > >> snapshot isolation using that mechanism. > > > > > >> > > > > > >> Paris > > > > > >> > > > > > >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> > > wrote: > > > > > >>> > > > > > >>> Hi All! > > > > > >>> > > > > > >>> I want to share with you a little project we have been working > on > > > at > > > > > King > > > > > >>> (with some help from some dataArtisans folks). I think this > would > > > be > > > > a > > > > > >>> valuable addition to Flink and solve a bunch of outstanding > > > > production > > > > > >>> use-cases and headaches around state bootstrapping and state > > > > analytics. > > > > > >>> > > > > > >>> We have built a quick and dirty POC implementation on top of > > Flink > > > > 1.6, > > > > > >>> please check the README for some nice examples to get a quick > > idea: > > > > > >>> > > > > > >>> https://github.com/king/bravo > > > > > >>> > > > > > >>> *Short story* > > > > > >>> Bravo is a convenient state reader and writer library > leveraging > > > the > > > > > >>> Flink’s batch processing capabilities. It supports processing > and > > > > > writing > > > > > >>> Flink streaming savepoints. At the moment it only supports > > > processing > > > > > >>> RocksDB savepoints but this can be extended in the future for > > other > > > > > state > > > > > >>> backends and checkpoint types. > > > > > >>> > > > > > >>> Our goal is to cover a few basic features: > > > > > >>> > > > > > >>> - Converting keyed states to Flink DataSets for processing and > > > > > >> analytics > > > > > >>> - Reading/Writing non-keyed operators states > > > > > >>> - Bootstrap keyed states from Flink DataSets and create new > > valid > > > > > >>> savepoints > > > > > >>> - Transform existing savepoints by replacing/changing some > > states > > > > > >>> > > > > > >>> > > > > > >>> Some example use-cases: > > > > > >>> > > > > > >>> - Point-in-time state analytics across all operators and keys > > > > > >>> - Bootstrap state of a streaming job from external resources > > such > > > as > > > > > >>> reading from database/filesystem > > > > > >>> - Validate and potentially repair corrupted state of a > streaming > > > job > > > > > >>> - Change max parallelism of a job > > > > > >>> > > > > > >>> > > > > > >>> Our main goal is to start working together with other Flink > > > > production > > > > > >>> users and make this something useful that can be part of Flink. > > So > > > if > > > > > you > > > > > >>> have use-cases please talk to us :) > > > > > >>> I have also started a google doc which contains a little bit > more > > > > info > > > > > >> than > > > > > >>> the readme and could be a starting place for discussions: > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > > > > >>> > > > > > >>> I know there are a bunch of rough edges and bugs (and no tests) > > but > > > > our > > > > > >>> motto is: If you are not embarrassed, you released too late :) > > > > > >>> > > > > > >>> Please let me know what you think! > > > > > >>> > > > > > >>> Cheers, > > > > > >>> Gyula > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -- > "So you have to trust that the dots will somehow connect in your future." > |
Hi Gyula,
+1 for this feature. State bootstrapping and state analytics will be very helpful. On Thu, Aug 23, 2018 at 4:09 AM Gyula Fóra <[hidden email]> wrote: > Hi Shuyi, > > The tool allows you to convert a Flink DataSet containing the individual > state rows, (key, state) pairs, into a state for a streaming operator. > > So if you want to bootstrap your state from data on HDFS, you would read > the file in Flink, convert it to the required DataSet format, then you can > write the state for the desired operator. > As you can read many different input formats in Flink this gives large > flexibility to where you can bootstrap the state from. > > Gyula > > Shuyi Chen <[hidden email]> ezt írta (időpont: 2018. aug. 22., Sze, > 21:28): > > > +1 on the tooling. Also, you mentioned about state bootstrapping problem. > > Could you please elaborate on how we can leverage the tooling to solve > > state bootstrapping? I think this is a common problem to stream > processing, > > and it will be great the community can work on it. Thanks. > > > > Shuyi > > > > On Wed, Aug 22, 2018 at 11:51 AM Gyula Fóra <[hidden email]> > wrote: > > > > > Thanks, > > > > > > I guess the first thing that would be great help from anyone interested > > in > > > helping is to try it for some streaming state :) > > > > > > We have tested these tools at King to analyze, transform and perform > some > > > aggregations on our user-states. The major limitation is that it > requires > > > RocksDB savepoints to work but other than that we successfully > analyzed a > > > few hundred gigabytes of state including reading keyed, and broadcast > > > states from different operators. Also you need to have a savepoint > before > > > you can create a new savepoint (with whatever state). > > > > > > Once we have some people who have played with it we can probably > greatly > > > improve the API and user experience as it is pretty low level at the > > > moment. I suggest we use the King git repo < > > https://github.com/king/bravo> > > > for > > > now to track some features before it is in a shape that deserves a > Flink > > > PR. We are super happy to take any improvements, code contributions > from > > > anyone so dont hesitate to reach out to me if you have some ideas. > > > > > > Gyula > > > > > > > > > Rong Rong <[hidden email]> ezt írta (időpont: 2018. aug. 22., > Sze, > > > 17:06): > > > > > > > +1. Being able to analyze the state is a huge operational advantage. > > > > Thanks Gyula for the POC and I would be very interested in > contributing > > > to > > > > the work. > > > > > > > > -- > > > > Rong > > > > > > > > On Tue, Aug 21, 2018 at 4:26 AM Till Rohrmann <[hidden email]> > > > > wrote: > > > > > > > > > big +1 for this feature. A tool to get your state out of and into > > Flink > > > > > will be tremendously helpful. > > > > > > > > > > On Mon, Aug 20, 2018 at 10:21 AM Aljoscha Krettek < > > [hidden email] > > > > > > > > > wrote: > > > > > > > > > > > +1 I'd like to have something like this in Flink a lot! > > > > > > > > > > > > > On 19. Aug 2018, at 11:57, Gyula Fóra <[hidden email]> > > > wrote: > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > Thanks for the feedback and I'm happy there is some interest :) > > > > > > > Tomorrow I will start improving the proposal based on the > > feedback > > > > and > > > > > > will > > > > > > > get back to work. > > > > > > > > > > > > > > If you are interested working together in this please ping me > and > > > we > > > > > can > > > > > > > discuss some ideas/plans and how to share work. > > > > > > > > > > > > > > Cheers, > > > > > > > Gyula > > > > > > > > > > > > > > Paris Carbone <[hidden email]> ezt írta (időpont: 2018. aug. > 18., > > > > Szo, > > > > > > 9:03): > > > > > > > > > > > > > >> +1 > > > > > > >> > > > > > > >> Might also be a good start to implement queryable stream state > > > with > > > > > > >> snapshot isolation using that mechanism. > > > > > > >> > > > > > > >> Paris > > > > > > >> > > > > > > >>> On 17 Aug 2018, at 12:28, Gyula Fóra <[hidden email]> > > > wrote: > > > > > > >>> > > > > > > >>> Hi All! > > > > > > >>> > > > > > > >>> I want to share with you a little project we have been > working > > on > > > > at > > > > > > King > > > > > > >>> (with some help from some dataArtisans folks). I think this > > would > > > > be > > > > > a > > > > > > >>> valuable addition to Flink and solve a bunch of outstanding > > > > > production > > > > > > >>> use-cases and headaches around state bootstrapping and state > > > > > analytics. > > > > > > >>> > > > > > > >>> We have built a quick and dirty POC implementation on top of > > > Flink > > > > > 1.6, > > > > > > >>> please check the README for some nice examples to get a quick > > > idea: > > > > > > >>> > > > > > > >>> https://github.com/king/bravo > > > > > > >>> > > > > > > >>> *Short story* > > > > > > >>> Bravo is a convenient state reader and writer library > > leveraging > > > > the > > > > > > >>> Flink’s batch processing capabilities. It supports processing > > and > > > > > > writing > > > > > > >>> Flink streaming savepoints. At the moment it only supports > > > > processing > > > > > > >>> RocksDB savepoints but this can be extended in the future for > > > other > > > > > > state > > > > > > >>> backends and checkpoint types. > > > > > > >>> > > > > > > >>> Our goal is to cover a few basic features: > > > > > > >>> > > > > > > >>> - Converting keyed states to Flink DataSets for processing > and > > > > > > >> analytics > > > > > > >>> - Reading/Writing non-keyed operators states > > > > > > >>> - Bootstrap keyed states from Flink DataSets and create new > > > valid > > > > > > >>> savepoints > > > > > > >>> - Transform existing savepoints by replacing/changing some > > > states > > > > > > >>> > > > > > > >>> > > > > > > >>> Some example use-cases: > > > > > > >>> > > > > > > >>> - Point-in-time state analytics across all operators and > keys > > > > > > >>> - Bootstrap state of a streaming job from external resources > > > such > > > > as > > > > > > >>> reading from database/filesystem > > > > > > >>> - Validate and potentially repair corrupted state of a > > streaming > > > > job > > > > > > >>> - Change max parallelism of a job > > > > > > >>> > > > > > > >>> > > > > > > >>> Our main goal is to start working together with other Flink > > > > > production > > > > > > >>> users and make this something useful that can be part of > Flink. > > > So > > > > if > > > > > > you > > > > > > >>> have use-cases please talk to us :) > > > > > > >>> I have also started a google doc which contains a little bit > > more > > > > > info > > > > > > >> than > > > > > > >>> the readme and could be a starting place for discussions: > > > > > > >>> > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/103k6wPX20kMu5H3SOOXSg5PZIaYpwdhqBMr-ppkFL5E/edit?usp=sharing > > > > > > >>> > > > > > > >>> I know there are a bunch of rough edges and bugs (and no > tests) > > > but > > > > > our > > > > > > >>> motto is: If you are not embarrassed, you released too late > :) > > > > > > >>> > > > > > > >>> Please let me know what you think! > > > > > > >>> > > > > > > >>> Cheers, > > > > > > >>> Gyula > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > "So you have to trust that the dots will somehow connect in your future." > > > |
Free forum by Nabble | Edit this page |