Hi,
I have a simple question on to how one could measure the individual execution times for an iteration step, without parsing the logs :) Is there a best practise for that? I'm interested in producing something like this http://data-artisans.com/img/blog/bulk_runtimes.png From the blog http://data-artisans.com/data-analysis-with-flink.html Johannes |
Hi,
Right now, you have to parse the logs (of the TaskManager running the Sync Task). We've created the linked images using this (very hacky) tool: https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java . I'm currently working on improving the monitoring, but realistically it will take some time until we expose iteration details in the web frontend. (Contributions are welcome as always ;) ) Best, Robert On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes < [hidden email]> wrote: > Hi, > > > I have a simple question on to how one could measure the individual > execution times for an iteration step, without parsing the logs :) > > > Is there a best practise for that? > > > I'm interested in producing something like this > > http://data-artisans.com/img/blog/bulk_runtimes.png > > > From the blog > > http://data-artisans.com/data-analysis-with-flink.html > > > > Johannes > |
Hi,
thanks for the quick answer - this tool might just do the job for the moment. Johannes ________________________________________ Von: Robert Metzger <[hidden email]> Gesendet: Samstag, 14. Februar 2015 16:38 An: [hidden email] Betreff: Re: Measuring Iteration Timings Hi, Right now, you have to parse the logs (of the TaskManager running the Sync Task). We've created the linked images using this (very hacky) tool: https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java . I'm currently working on improving the monitoring, but realistically it will take some time until we expose iteration details in the web frontend. (Contributions are welcome as always ;) ) Best, Robert On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes < [hidden email]> wrote: > Hi, > > > I have a simple question on to how one could measure the individual > execution times for an iteration step, without parsing the logs :) > > > Is there a best practise for that? > > > I'm interested in producing something like this > > http://data-artisans.com/img/blog/bulk_runtimes.png > > > From the blog > > http://data-artisans.com/data-analysis-with-flink.html > > > > Johannes > |
Johannes!
You can also customize your messages by printing a message to standard out in the open() and close() methods of your functions. You have to extend the RichFunction in that case. Greetings, Stephan On Sat, Feb 14, 2015 at 4:47 PM, Kirschnick, Johannes < [hidden email]> wrote: > Hi, > > thanks for the quick answer - this tool might just do the job for the > moment. > > Johannes > ________________________________________ > Von: Robert Metzger <[hidden email]> > Gesendet: Samstag, 14. Februar 2015 16:38 > An: [hidden email] > Betreff: Re: Measuring Iteration Timings > > Hi, > > Right now, you have to parse the logs (of the TaskManager running the Sync > Task). > We've created the linked images using this (very hacky) tool: > > https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java > . > > I'm currently working on improving the monitoring, but realistically it > will take some time until we expose iteration details in the web frontend. > (Contributions are welcome as always ;) ) > > Best, > Robert > > > > On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes < > [hidden email]> wrote: > > > Hi, > > > > > > I have a simple question on to how one could measure the individual > > execution times for an iteration step, without parsing the logs :) > > > > > > Is there a best practise for that? > > > > > > I'm interested in producing something like this > > > > http://data-artisans.com/img/blog/bulk_runtimes.png > > > > > > From the blog > > > > http://data-artisans.com/data-analysis-with-flink.html > > > > > > > > Johannes > > > |
Free forum by Nabble | Edit this page |