Hi
As Flink streaming API is being stabilized, it is worth investigating tooling around it to make developer's life easier and also turning Flink streaming a more attractive stream processing platform. I would like to suggest a couple tools which are worth integrating with Flink streaming: 1. 'StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows. The ultimate goal of StreamFlow is to make working with stream processing frameworks such as Apache Storm easier, faster, and with "enterprise" like management functionality. StreamFlow also provides a mechanism for non-developers such as data scientists, analysts, or operational users to rapidly build scalable data flows and analytics.' Reference:https://github.com/lmco/streamflow The topology builder looks really slick! Any chance of seeing StreamFlow a GUI for building Flink streaming data flows? 2. SPQR pronounced Spooker is a framework for building dynamic data stream processing pipelines: a feature that is neither available in Apache Storm nor in Spark Streaming. https://github.com/ottogroup/SPQR It might be worth investigating SPQR, integrating it with Flink and make it production ready. What are your thoughts? Thanks Slim Baltagi |
Hi Slim,
I totally agree with you that we should start working on supporting tools like these. StreamFlow is really nice tool, I also like it a lot. I think it would be not too hard to integrate it with Flink as there are several projects that have already used Flink streaming in a compositional way. (SAMOA, or the Storm compatibility layer). So I guess this is just a matter of someone picking it up at one point, but I am really looking forward to it , and will support anyone doing it :) As for SPQR and the dynamic pipelines, I don't think we have this on the short term roadmap, but it would indeed be a nice feature. I don't see any system limitations that would keep us from implementing this either once we start working on dynamic modifications to the execution graph (such as for automatic scaling) Cheers, Gyula Slim Baltagi <[hidden email]> ezt írta (időpont: 2015. júl. 30., Cs, 5:50): > Hi > > As Flink streaming API is being stabilized, it is worth investigating > tooling around it to make developer's life easier and also turning Flink > streaming a more attractive stream processing platform. > I would like to suggest a couple tools which are worth integrating with > Flink streaming: > > 1. 'StreamFlow™ is a stream processing tool designed to help build and > monitor processing workflows. The ultimate goal of StreamFlow is to make > working with stream processing frameworks such as Apache Storm easier, > faster, and with "enterprise" like management functionality. > > StreamFlow also provides a mechanism for non-developers such as data > scientists, analysts, or operational users to rapidly build scalable data > flows and analytics.' > > Reference:https://github.com/lmco/streamflow The topology builder looks > really slick! > > Any chance of seeing StreamFlow a GUI for building Flink streaming data > flows? > > 2. SPQR pronounced Spooker is a framework for building *dynamic * data > stream processing pipelines: a feature that is neither available in Apache > Storm nor in Spark Streaming. > https://github.com/ottogroup/SPQR > > It might be worth investigating SPQR, integrating it with Flink and make it > production ready. > > What are your thoughts? > > Thanks > > Slim Baltagi > > > > -- > View this message in context: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/On-some-GUI-tools-for-building-Flink-Streaming-data-flows-tp7233.html > Sent from the Apache Flink Mailing List archive. mailing list archive at > Nabble.com. > |
Hi Gyula, hi Slim,
being the core developer of SPQR, I am very glad to see the framework referenced on the Apache Flink dev list ;-) Following the same approach as Flink - using real data streaming instead of mini-batching - SPQR adds some ad hoc'ness features to introduce more flexibility into the streaming model. The core idea is to shield away complex topology implementation and deployments but provide a more business oriented layer for ad hoc queries and ad hoc stream modificiation. From our point of view both features prepare stream processing for usage within contexts that are less tech oriented and where SQL is a broadly spoken language. But .. it is not about refactoring SQL to stream processing (as I am not sure if a language specified for static data is suitable for streaming data) but to find a suitable approach which is somehow similar to SQL when it comes to adopting it in a business oriented context. The most accurate metaphor for describing how we expect stream processing to look like for its users is that of a plumber working on a (more or less) large pipe system. As both approaches are quite similar but Flink already provides some very ice features that SPQR lacks, eg graph optimization & distribution and cluster support, I think it may be worth checking if porting our vision of ad hoc'ness into Flink is possible - maybe as a layer on top of the framework or as more core integration. Therefore - if such a feature is of any interest for you - I would like to offer support on that task. As outlined above and as Slim also mentioned it regarding StreamFlow UI providing a less tech oriented but more business oriented layer may lay ground for a broad adoption of Flink since NONE of the existing frameworks provides such a feature. All the other kids on the block require heavy code compilation and deployment before any work is done. Cheers, Christian (@mnxfst) |
Hey Christian,
Thanks for the insider view on SPQR. I have to agree with Gyula that dynamic topology build is not the highest priority for Flink currently, but certainly a very interesting feature and one that has already been requested by a couple of users. As for none of the open source streaming systems supporting a UI: with the open sourcing project Apex Datatorrent definitely has merit in this field, although I have to admit I do not exactly know how much of their platform is open sourced currently. Best, Marton On Tue, Aug 11, 2015 at 12:04 PM, mnxfst <[hidden email]> wrote: > mnxfst |
Hi Christian!
Sounds like a very cool proposal, and it seems that Flink and SPQR could complement each other very well. If you are interested to see this on top of Flink, let's have a chat to see how to best get started with this - what would be the easiest points of integration. As said, Flink cannot support dynamic topology changes at this point, but interestingly, a lot of the building blocks for dynamic topology growths are in place in the lower runtime levels, so even that could be added in the future. Greetings, Stephan On Tue, Aug 11, 2015 at 12:44 PM, Márton Balassi <[hidden email]> wrote: > Hey Christian, > > Thanks for the insider view on SPQR. I have to agree with Gyula that > dynamic topology build is not the highest priority for Flink currently, but > certainly a very interesting feature and one that has already been > requested by a couple of users. > > As for none of the open source streaming systems supporting a UI: with the > open sourcing project Apex Datatorrent definitely has merit in this field, > although I have to admit I do not exactly know how much of their platform > is open sourced currently. > > Best, > > Marton > > > On Tue, Aug 11, 2015 at 12:04 PM, mnxfst <[hidden email]> wrote: > > > mnxfst > |
Free forum by Nabble | Edit this page |