Measuring Iteration Timings

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Measuring Iteration Timings

Kirschnick, Johannes
Hi,


I have a simple question on to how one could measure the individual execution times for an iteration step, without parsing the logs :)


Is there a best practise for that?


I'm interested in producing something like this

http://data-artisans.com/img/blog/bulk_runtimes.png


From the blog

http://data-artisans.com/data-analysis-with-flink.html



Johannes
Reply | Threaded
Open this post in threaded view
|

Re: Measuring Iteration Timings

Robert Metzger
Hi,

Right now, you have to parse the logs (of the TaskManager running the Sync
Task).
We've created the linked images using this (very hacky) tool:
https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java
.

I'm currently working on improving the monitoring, but realistically it
will take some time until we expose iteration details in the web frontend.
(Contributions are welcome as always ;) )

Best,
Robert



On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes <
[hidden email]> wrote:

> Hi,
>
>
> I have a simple question on to how one could measure the individual
> execution times for an iteration step, without parsing the logs :)
>
>
> Is there a best practise for that?
>
>
> I'm interested in producing something like this
>
> http://data-artisans.com/img/blog/bulk_runtimes.png
>
>
> From the blog
>
> http://data-artisans.com/data-analysis-with-flink.html
>
>
>
> Johannes
>
Reply | Threaded
Open this post in threaded view
|

Re: Measuring Iteration Timings

Kirschnick, Johannes
Hi,

thanks for the quick answer - this tool might just do the job for the moment.

Johannes
________________________________________
Von: Robert Metzger <[hidden email]>
Gesendet: Samstag, 14. Februar 2015 16:38
An: [hidden email]
Betreff: Re: Measuring Iteration Timings

Hi,

Right now, you have to parse the logs (of the TaskManager running the Sync
Task).
We've created the linked images using this (very hacky) tool:
https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java
.

I'm currently working on improving the monitoring, but realistically it
will take some time until we expose iteration details in the web frontend.
(Contributions are welcome as always ;) )

Best,
Robert



On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes <
[hidden email]> wrote:

> Hi,
>
>
> I have a simple question on to how one could measure the individual
> execution times for an iteration step, without parsing the logs :)
>
>
> Is there a best practise for that?
>
>
> I'm interested in producing something like this
>
> http://data-artisans.com/img/blog/bulk_runtimes.png
>
>
> From the blog
>
> http://data-artisans.com/data-analysis-with-flink.html
>
>
>
> Johannes
>
Reply | Threaded
Open this post in threaded view
|

Re: Measuring Iteration Timings

Stephan Ewen
Johannes!

You can also customize your messages by printing a message to standard out
in the open() and close() methods of your functions. You have to extend the
RichFunction in that case.

Greetings,
Stephan


On Sat, Feb 14, 2015 at 4:47 PM, Kirschnick, Johannes <
[hidden email]> wrote:

> Hi,
>
> thanks for the quick answer - this tool might just do the job for the
> moment.
>
> Johannes
> ________________________________________
> Von: Robert Metzger <[hidden email]>
> Gesendet: Samstag, 14. Februar 2015 16:38
> An: [hidden email]
> Betreff: Re: Measuring Iteration Timings
>
> Hi,
>
> Right now, you have to parse the logs (of the TaskManager running the Sync
> Task).
> We've created the linked images using this (very hacky) tool:
>
> https://github.com/project-flink/flink-perf/blob/master/flink-jobs/src/main/java/com/github/projectflink/utils/IterationParser.java
> .
>
> I'm currently working on improving the monitoring, but realistically it
> will take some time until we expose iteration details in the web frontend.
> (Contributions are welcome as always ;) )
>
> Best,
> Robert
>
>
>
> On Sat, Feb 14, 2015 at 4:31 PM, Kirschnick, Johannes <
> [hidden email]> wrote:
>
> > Hi,
> >
> >
> > I have a simple question on to how one could measure the individual
> > execution times for an iteration step, without parsing the logs :)
> >
> >
> > Is there a best practise for that?
> >
> >
> > I'm interested in producing something like this
> >
> > http://data-artisans.com/img/blog/bulk_runtimes.png
> >
> >
> > From the blog
> >
> > http://data-artisans.com/data-analysis-with-flink.html
> >
> >
> >
> > Johannes
> >
>