Hi list,
I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools) So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file The results look like this... http://scaleunlimited.com/downloads/CrawlTopology.png If this is useful (and doesn’t already exist) then I can easily drop it in somewhere. — Ken -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr |
Hi Ken,
Maybe you can look into this one : http://flink.apache.org/visualizer/. - Pat > On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote: > > Hi list, > > I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools) > > So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file > > The results look like this... > > http://scaleunlimited.com/downloads/CrawlTopology.png > > If this is useful (and doesn’t already exist) then I can easily drop it in somewhere. > > — Ken > > -------------------------- > Ken Krugler > +1 530-210-6378 > http://www.scaleunlimited.com > custom big data solutions & training > Hadoop, Cascading, Cassandra & Solr > > > |
Hey Ken!
This looks really good. +1 to make this available publicly. We can link it from the Flink website and the viz tool Pat linked to. The vizualizer has currently some open issues, it is not up to date with the one that is part of the Flink web UI. – Ufuk On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai <[hidden email]> wrote: > Hi Ken, > > Maybe you can look into this one : http://flink.apache.org/visualizer/. > > - Pat > >> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote: >> >> Hi list, >> >> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools) >> >> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file >> >> The results look like this... >> >> http://scaleunlimited.com/downloads/CrawlTopology.png >> >> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere. >> >> — Ken >> >> -------------------------- >> Ken Krugler >> +1 530-210-6378 >> http://www.scaleunlimited.com >> custom big data solutions & training >> Hadoop, Cascading, Cassandra & Solr >> >> >> > |
In reply to this post by Pattarawat Chormai
Hi Pat,
> On Feb 21, 2017, at 6:01pm, Pattarawat Chormai <[hidden email]> wrote: > > Hi Ken, > > Maybe you can look into this one : http://flink.apache.org/visualizer/. Thanks, that’s interesting and convenient. Though I’d probably keep using OmniGraffle with a dot file as that gives me the ability to edit/annotate as needed. Thoughts on having the iteration sink connect to the iteration source? I found that helpful when working on my topology. Regards, — Ken >> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote: >> >> Hi list, >> >> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools) >> >> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file >> >> The results look like this... >> >> http://scaleunlimited.com/downloads/CrawlTopology.png >> >> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere. >> >> — Ken -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr |
In reply to this post by Ufuk Celebi-2
Hi Ufuk,
> On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote: > > Hey Ken! > > This looks really good. +1 to make this available publicly. > > We can link it from the Flink website and the viz tool Pat linked to. > The vizualizer has currently some open issues, it is not up to date > with the one that is part of the Flink web UI. I wasn’t planning on running a web interface to the tool. My thought was to just add a writeDotFile(“path to file") method to the StreamExecutionEnvironment…this is similar to what Cascading supports. If that’s reasonable, I’ll open an issue and attach the code. — Ken > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai > <[hidden email]> wrote: >> Hi Ken, >> >> Maybe you can look into this one : http://flink.apache.org/visualizer/. >> >> - Pat >> >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote: >>> >>> Hi list, >>> >>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools) >>> >>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file >>> >>> The results look like this... >>> >>> http://scaleunlimited.com/downloads/CrawlTopology.png >>> >>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere. >>> >>> — Ken -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr |
Hi Ken,
I think this would be an interesting feature! I'd suggest to open a JIRA for it. When extending the API of core classes such as ExecutionEnvironment, there is often some discussion whether the feature is important enough or whether it should be rather added to some external util class (which makes it obviously less visible). I think this feature would be valuable enough to be added directly into the ExecutionEnvironment. DOT is a common format for graph and flow visualizations. Let's see what others think. Best, Fabian 2017-02-24 0:15 GMT+01:00 Ken Krugler <[hidden email]>: > Hi Ufuk, > > > On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote: > > > > Hey Ken! > > > > This looks really good. +1 to make this available publicly. > > > > We can link it from the Flink website and the viz tool Pat linked to. > > The vizualizer has currently some open issues, it is not up to date > > with the one that is part of the Flink web UI. > > I wasn’t planning on running a web interface to the tool. > > My thought was to just add a writeDotFile(“path to file") method to the > StreamExecutionEnvironment…this is similar to what Cascading supports. > > If that’s reasonable, I’ll open an issue and attach the code. > > — Ken > > > > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai > > <[hidden email]> wrote: > >> Hi Ken, > >> > >> Maybe you can look into this one : http://flink.apache.org/visualizer/. > >> > >> - Pat > >> > >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> > wrote: > >>> > >>> Hi list, > >>> > >>> I poked around a bit and didn’t see a way to easily visualize > workflows in the same way I was used to with Cascading (generate a blah.dot > file, open with OminGraffle or other graph tools) > >>> > >>> So I wrote a bit of code that takes the JSON output from > StreamExecutionEnvironment#getExecutionPlan and generates a .dot file > >>> > >>> The results look like this... > >>> > >>> http://scaleunlimited.com/downloads/CrawlTopology.png > >>> > >>> If this is useful (and doesn’t already exist) then I can easily drop > it in somewhere. > >>> > >>> — Ken > > -------------------------- > Ken Krugler > +1 530-210-6378 > http://www.scaleunlimited.com > custom big data solutions & training > Hadoop, Cascading, Cassandra & Solr > > > > |
Ken and Fabian,
Is the use case to generate and act on the dot file from within the user program? Would it be more maintainable to make the plan JSON more accessible (through the CLI and web interface) which users could then pipe through a converter script? Greg On Fri, Feb 24, 2017 at 4:55 AM, Fabian Hueske <[hidden email]> wrote: > Hi Ken, > > I think this would be an interesting feature! > I'd suggest to open a JIRA for it. > > When extending the API of core classes such as ExecutionEnvironment, there > is often some discussion whether the feature is important enough or whether > it should be rather added to some external util class (which makes it > obviously less visible). > I think this feature would be valuable enough to be added directly into the > ExecutionEnvironment. DOT is a common format for graph and flow > visualizations. > > Let's see what others think. > > Best, Fabian > > > 2017-02-24 0:15 GMT+01:00 Ken Krugler <[hidden email]>: > > > Hi Ufuk, > > > > > On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote: > > > > > > Hey Ken! > > > > > > This looks really good. +1 to make this available publicly. > > > > > > We can link it from the Flink website and the viz tool Pat linked to. > > > The vizualizer has currently some open issues, it is not up to date > > > with the one that is part of the Flink web UI. > > > > I wasn’t planning on running a web interface to the tool. > > > > My thought was to just add a writeDotFile(“path to file") method to the > > StreamExecutionEnvironment…this is similar to what Cascading supports. > > > > If that’s reasonable, I’ll open an issue and attach the code. > > > > — Ken > > > > > > > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai > > > <[hidden email]> wrote: > > >> Hi Ken, > > >> > > >> Maybe you can look into this one : http://flink.apache.org/ > visualizer/. > > >> > > >> - Pat > > >> > > >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler < > [hidden email]> > > wrote: > > >>> > > >>> Hi list, > > >>> > > >>> I poked around a bit and didn’t see a way to easily visualize > > workflows in the same way I was used to with Cascading (generate a > blah.dot > > file, open with OminGraffle or other graph tools) > > >>> > > >>> So I wrote a bit of code that takes the JSON output from > > StreamExecutionEnvironment#getExecutionPlan and generates a .dot file > > >>> > > >>> The results look like this... > > >>> > > >>> http://scaleunlimited.com/downloads/CrawlTopology.png > > >>> > > >>> If this is useful (and doesn’t already exist) then I can easily drop > > it in somewhere. > > >>> > > >>> — Ken > > > > -------------------------- > > Ken Krugler > > +1 530-210-6378 > > http://www.scaleunlimited.com > > custom big data solutions & training > > Hadoop, Cascading, Cassandra & Solr > > > > > > > > > |
Hi Greg,
The use case is to create a visualization of the topology. So I don’t think there’s any reason to “act on the dot file from within the user program” Regards, — Ken > On Feb 24, 2017, at 7:51am, Greg Hogan <[hidden email]> wrote: > > Ken and Fabian, > > Is the use case to generate and act on the dot file from within the user > program? Would it be more maintainable to make the plan JSON more > accessible (through the CLI and web interface) which users could then pipe > through a converter script? > > Greg > > On Fri, Feb 24, 2017 at 4:55 AM, Fabian Hueske <[hidden email]> wrote: > >> Hi Ken, >> >> I think this would be an interesting feature! >> I'd suggest to open a JIRA for it. >> >> When extending the API of core classes such as ExecutionEnvironment, there >> is often some discussion whether the feature is important enough or whether >> it should be rather added to some external util class (which makes it >> obviously less visible). >> I think this feature would be valuable enough to be added directly into the >> ExecutionEnvironment. DOT is a common format for graph and flow >> visualizations. >> >> Let's see what others think. >> >> Best, Fabian >> >> >> 2017-02-24 0:15 GMT+01:00 Ken Krugler <[hidden email]>: >> >>> Hi Ufuk, >>> >>>> On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote: >>>> >>>> Hey Ken! >>>> >>>> This looks really good. +1 to make this available publicly. >>>> >>>> We can link it from the Flink website and the viz tool Pat linked to. >>>> The vizualizer has currently some open issues, it is not up to date >>>> with the one that is part of the Flink web UI. >>> >>> I wasn’t planning on running a web interface to the tool. >>> >>> My thought was to just add a writeDotFile(“path to file") method to the >>> StreamExecutionEnvironment…this is similar to what Cascading supports. >>> >>> If that’s reasonable, I’ll open an issue and attach the code. >>> >>> — Ken >>> >>> >>>> On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai >>>> <[hidden email]> wrote: >>>>> Hi Ken, >>>>> >>>>> Maybe you can look into this one : http://flink.apache.org/ >> visualizer/. >>>>> >>>>> - Pat >>>>> >>>>>> On Feb 21, 2017, at 11:59 PM, Ken Krugler < >> [hidden email]> >>> wrote: >>>>>> >>>>>> Hi list, >>>>>> >>>>>> I poked around a bit and didn’t see a way to easily visualize >>> workflows in the same way I was used to with Cascading (generate a >> blah.dot >>> file, open with OminGraffle or other graph tools) >>>>>> >>>>>> So I wrote a bit of code that takes the JSON output from >>> StreamExecutionEnvironment#getExecutionPlan and generates a .dot file >>>>>> >>>>>> The results look like this... >>>>>> >>>>>> http://scaleunlimited.com/downloads/CrawlTopology.png >>>>>> >>>>>> If this is useful (and doesn’t already exist) then I can easily drop >>> it in somewhere. >>>>>> >>>>>> — Ken >>> >>> -------------------------- >>> Ken Krugler >>> +1 530-210-6378 >>> http://www.scaleunlimited.com >>> custom big data solutions & training >>> Hadoop, Cascading, Cassandra & Solr >>> >>> >>> >>> >> -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr |
Free forum by Nabble | Edit this page |