Hello community,
For almost a year in data Artisans Nico and I were maintaining a setup that continuously evaluates Flink with benchmarks defined at https://github.com/dataArtisans/flink-benchmarks <https://github.com/dataArtisans/flink-benchmarks>. With growing interest and after proving useful a couple of times, we have finally decided to publish the web UI layer of this setup. Currently it is accessible via the following (maybe not so?) temporarily url: http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> This is a simple web UI to present performance changes over past and present commits to Apache Flink. It only has a couple of views and the most useful ones are: 1. Timeline 2. Comparison (I recommend to use normalization) Timeline is useful for spotting unintended regressions or unexpected improvements. It is being updated every six hours. Comparison is useful for comparing a given branch (for example a pending PR) with the master branch. More about that later. The codespeed project on it’s own is just a presentation layer. As mentioned before, the only currently available benchmarks are defined in the flink-benchmarks repository and they are executed periodically or on demand by Jenkins on a single bare metal machine. The current setup limits us only to micro benchmarks (they are easier to setup/develop/maintain and have a quicker feedback loop compared to cluster benchmarks) but there is no reason preventing us from setting up other kinds of benchmarks and upload their results to our codespeed instance as well. Regarding the comparison view. Currently data Artisans’ Flink mirror repository at https://github.com/dataArtisans/flink <https://github.com/dataArtisans/flink> is configured to trigger benchmark runs on every commit/change that happens on the benchmark-request branch (We chose to use dataArtisans' repository here because we needed a custom GitHub hook that we couldn’t add to the apache/flink repository). Benchmarking usually takes between one and two hours. One obvious limitation at the moment is that there is only one comparison view, with one comparison branch, so trying to compare two PRs at the same time is impossible. However we can tackle this problem once it will become a real issue, not only a theoretical one. Piotrek & Nico |
Thanks for this contribution Piotr and Nico.
Tools like this are really useful for Flink’s success. Cheers, Kostas > On Sep 21, 2018, at 4:59 PM, Piotr Nowojski <[hidden email]> wrote: > > Hello community, > > For almost a year in data Artisans Nico and I were maintaining a setup > that continuously evaluates Flink with benchmarks defined at > https://github.com/dataArtisans/flink-benchmarks <https://github.com/dataArtisans/flink-benchmarks>. With growing interest > and after proving useful a couple of times, we have finally decided to > publish the web UI layer of this setup. Currently it is accessible via > the following (maybe not so?) temporarily url: > > http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> > > This is a simple web UI to present performance changes over past and > present commits to Apache Flink. It only has a couple of views and the > most useful ones are: > > 1. Timeline > 2. Comparison (I recommend to use normalization) > > Timeline is useful for spotting unintended regressions or unexpected > improvements. It is being updated every six hours. > Comparison is useful for comparing a given branch (for example a pending > PR) with the master branch. More about that later. > > The codespeed project on it’s own is just a presentation layer. As > mentioned before, the only currently available benchmarks are defined in > the flink-benchmarks repository and they are executed periodically or on > demand by Jenkins on a single bare metal machine. The current setup > limits us only to micro benchmarks (they are easier to > setup/develop/maintain and have a quicker feedback loop compared to > cluster benchmarks) but there is no reason preventing us from setting up > other kinds of benchmarks and upload their results to our codespeed > instance as well. > > Regarding the comparison view. Currently data Artisans’ Flink mirror > repository at https://github.com/dataArtisans/flink <https://github.com/dataArtisans/flink> is configured to > trigger benchmark runs on every commit/change that happens on the > benchmark-request branch (We chose to use dataArtisans' repository here > because we needed a custom GitHub hook that we couldn’t add to the > apache/flink repository). Benchmarking usually takes between one and two > hours. One obvious limitation at the moment is that there is only one > comparison view, with one comparison branch, so trying to compare two > PRs at the same time is impossible. However we can tackle > this problem once it will become a real issue, not only a theoretical one. > > Piotrek & Nico |
Hi Piotr and Nico,
Thank you for your hard work, it looks great. With it, there will be quantifiable indicators of the impact of changes and improvements in Flink source code. Thanks, vino. Kostas Kloudas <[hidden email]> 于2018年9月22日周六 上午12:01写道: > Thanks for this contribution Piotr and Nico. > > Tools like this are really useful for Flink’s success. > > Cheers, > Kostas > > > On Sep 21, 2018, at 4:59 PM, Piotr Nowojski <[hidden email]> > wrote: > > > > Hello community, > > > > For almost a year in data Artisans Nico and I were maintaining a setup > > that continuously evaluates Flink with benchmarks defined at > > https://github.com/dataArtisans/flink-benchmarks < > https://github.com/dataArtisans/flink-benchmarks>. With growing interest > > and after proving useful a couple of times, we have finally decided to > > publish the web UI layer of this setup. Currently it is accessible via > > the following (maybe not so?) temporarily url: > > > > http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> > > > > This is a simple web UI to present performance changes over past and > > present commits to Apache Flink. It only has a couple of views and the > > most useful ones are: > > > > 1. Timeline > > 2. Comparison (I recommend to use normalization) > > > > Timeline is useful for spotting unintended regressions or unexpected > > improvements. It is being updated every six hours. > > Comparison is useful for comparing a given branch (for example a pending > > PR) with the master branch. More about that later. > > > > The codespeed project on it’s own is just a presentation layer. As > > mentioned before, the only currently available benchmarks are defined in > > the flink-benchmarks repository and they are executed periodically or on > > demand by Jenkins on a single bare metal machine. The current setup > > limits us only to micro benchmarks (they are easier to > > setup/develop/maintain and have a quicker feedback loop compared to > > cluster benchmarks) but there is no reason preventing us from setting up > > other kinds of benchmarks and upload their results to our codespeed > > instance as well. > > > > Regarding the comparison view. Currently data Artisans’ Flink mirror > > repository at https://github.com/dataArtisans/flink < > https://github.com/dataArtisans/flink> is configured to > > trigger benchmark runs on every commit/change that happens on the > > benchmark-request branch (We chose to use dataArtisans' repository here > > because we needed a custom GitHub hook that we couldn’t add to the > > apache/flink repository). Benchmarking usually takes between one and two > > hours. One obvious limitation at the moment is that there is only one > > comparison view, with one comparison branch, so trying to compare two > > PRs at the same time is impossible. However we can tackle > > this problem once it will become a real issue, not only a theoretical > one. > > > > Piotrek & Nico > > |
In reply to this post by Piotr Nowojski
Thanks @Piotr Nowojski and @Nico Kruber for the good job!
I already benefit from this benchmark in the previous PRs. Wish the visualization tool becoming stronger to benefit more for the community! Best, Zhijiang ------------------------------------------------------------------ 发件人:Piotr Nowojski <[hidden email]> 发送时间:2018年9月21日(星期五) 22:59 收件人:dev <[hidden email]> 抄 送:Nico Kruber <[hidden email]> 主 题:Codespeed deployment for Flink Hello community, For almost a year in data Artisans Nico and I were maintaining a setup that continuously evaluates Flink with benchmarks defined at https://github.com/dataArtisans/flink-benchmarks <https://github.com/dataArtisans/flink-benchmarks>. With growing interest and after proving useful a couple of times, we have finally decided to publish the web UI layer of this setup. Currently it is accessible via the following (maybe not so?) temporarily url: http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> This is a simple web UI to present performance changes over past and present commits to Apache Flink. It only has a couple of views and the most useful ones are: 1. Timeline 2. Comparison (I recommend to use normalization) Timeline is useful for spotting unintended regressions or unexpected improvements. It is being updated every six hours. Comparison is useful for comparing a given branch (for example a pending PR) with the master branch. More about that later. The codespeed project on it’s own is just a presentation layer. As mentioned before, the only currently available benchmarks are defined in the flink-benchmarks repository and they are executed periodically or on demand by Jenkins on a single bare metal machine. The current setup limits us only to micro benchmarks (they are easier to setup/develop/maintain and have a quicker feedback loop compared to cluster benchmarks) but there is no reason preventing us from setting up other kinds of benchmarks and upload their results to our codespeed instance as well. Regarding the comparison view. Currently data Artisans’ Flink mirror repository at https://github.com/dataArtisans/flink <https://github.com/dataArtisans/flink> is configured to trigger benchmark runs on every commit/change that happens on the benchmark-request branch (We chose to use dataArtisans' repository here because we needed a custom GitHub hook that we couldn’t add to the apache/flink repository). Benchmarking usually takes between one and two hours. One obvious limitation at the moment is that there is only one comparison view, with one comparison branch, so trying to compare two PRs at the same time is impossible. However we can tackle this problem once it will become a real issue, not only a theoretical one. Piotrek & Nico |
Great tool!
> On Sep 24, 2018, at 10:59 PM, Zhijiang(wangzhijiang999) <[hidden email]> wrote: > > Thanks @Piotr Nowojski and @Nico Kruber for the good job! > > I already benefit from this benchmark in the previous PRs. Wish the visualization tool becoming stronger to benefit more for the community! > > Best, > Zhijiang > ------------------------------------------------------------------ > 发件人:Piotr Nowojski <[hidden email]> > 发送时间:2018年9月21日(星期五) 22:59 > 收件人:dev <[hidden email]> > 抄 送:Nico Kruber <[hidden email]> > 主 题:Codespeed deployment for Flink > > Hello community, > > For almost a year in data Artisans Nico and I were maintaining a setup > that continuously evaluates Flink with benchmarks defined at > https://github.com/dataArtisans/flink-benchmarks <https://github.com/dataArtisans/flink-benchmarks>. With growing interest > and after proving useful a couple of times, we have finally decided to > publish the web UI layer of this setup. Currently it is accessible via > the following (maybe not so?) temporarily url: > > http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> > > This is a simple web UI to present performance changes over past and > present commits to Apache Flink. It only has a couple of views and the > most useful ones are: > > 1. Timeline > 2. Comparison (I recommend to use normalization) > > Timeline is useful for spotting unintended regressions or unexpected > improvements. It is being updated every six hours. > Comparison is useful for comparing a given branch (for example a pending > PR) with the master branch. More about that later. > > The codespeed project on it’s own is just a presentation layer. As > mentioned before, the only currently available benchmarks are defined in > the flink-benchmarks repository and they are executed periodically or on > demand by Jenkins on a single bare metal machine. The current setup > limits us only to micro benchmarks (they are easier to > setup/develop/maintain and have a quicker feedback loop compared to > cluster benchmarks) but there is no reason preventing us from setting up > other kinds of benchmarks and upload their results to our codespeed > instance as well. > > Regarding the comparison view. Currently data Artisans’ Flink mirror > repository at https://github.com/dataArtisans/flink <https://github.com/dataArtisans/flink> is configured to > trigger benchmark runs on every commit/change that happens on the > benchmark-request branch (We chose to use dataArtisans' repository here > because we needed a custom GitHub hook that we couldn’t add to the > apache/flink repository). Benchmarking usually takes between one and two > hours. One obvious limitation at the moment is that there is only one > comparison view, with one comparison branch, so trying to compare two > PRs at the same time is impossible. However we can tackle > this problem once it will become a real issue, not only a theoretical one. > > Piotrek & Nico > |
It is a great tool. Thanks for the contribution.
On Tue, Sep 25, 2018 at 11:39 AM Jin Sun <[hidden email]> wrote: > Great tool! > > > On Sep 24, 2018, at 10:59 PM, Zhijiang(wangzhijiang999) < > [hidden email]> wrote: > > > > Thanks @Piotr Nowojski and @Nico Kruber for the good job! > > > > I already benefit from this benchmark in the previous PRs. Wish the > visualization tool becoming stronger to benefit more for the community! > > > > Best, > > Zhijiang > > ------------------------------------------------------------------ > > 发件人:Piotr Nowojski <[hidden email]> > > 发送时间:2018年9月21日(星期五) 22:59 > > 收件人:dev <[hidden email]> > > 抄 送:Nico Kruber <[hidden email]> > > 主 题:Codespeed deployment for Flink > > > > Hello community, > > > > For almost a year in data Artisans Nico and I were maintaining a setup > > that continuously evaluates Flink with benchmarks defined at > > https://github.com/dataArtisans/flink-benchmarks < > https://github.com/dataArtisans/flink-benchmarks>. With growing interest > > and after proving useful a couple of times, we have finally decided to > > publish the web UI layer of this setup. Currently it is accessible via > > the following (maybe not so?) temporarily url: > > > > http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> > > > > This is a simple web UI to present performance changes over past and > > present commits to Apache Flink. It only has a couple of views and the > > most useful ones are: > > > > 1. Timeline > > 2. Comparison (I recommend to use normalization) > > > > Timeline is useful for spotting unintended regressions or unexpected > > improvements. It is being updated every six hours. > > Comparison is useful for comparing a given branch (for example a pending > > PR) with the master branch. More about that later. > > > > The codespeed project on it’s own is just a presentation layer. As > > mentioned before, the only currently available benchmarks are defined in > > the flink-benchmarks repository and they are executed periodically or on > > demand by Jenkins on a single bare metal machine. The current setup > > limits us only to micro benchmarks (they are easier to > > setup/develop/maintain and have a quicker feedback loop compared to > > cluster benchmarks) but there is no reason preventing us from setting up > > other kinds of benchmarks and upload their results to our codespeed > > instance as well. > > > > Regarding the comparison view. Currently data Artisans’ Flink mirror > > repository at https://github.com/dataArtisans/flink < > https://github.com/dataArtisans/flink> is configured to > > trigger benchmark runs on every commit/change that happens on the > > benchmark-request branch (We chose to use dataArtisans' repository here > > because we needed a custom GitHub hook that we couldn’t add to the > > apache/flink repository). Benchmarking usually takes between one and two > > hours. One obvious limitation at the moment is that there is only one > > comparison view, with one comparison branch, so trying to compare two > > PRs at the same time is impossible. However we can tackle > > this problem once it will become a real issue, not only a theoretical > one. > > > > Piotrek & Nico > > > > |
Great addition Piotr and Nico. This is a really nice tool for the community
to monitor performance regressions in Flink. On Tue, Sep 25, 2018 at 8:41 PM Peter Huang <[hidden email]> wrote: > It is a great tool. Thanks for the contribution. > > On Tue, Sep 25, 2018 at 11:39 AM Jin Sun <[hidden email]> wrote: > > > Great tool! > > > > > On Sep 24, 2018, at 10:59 PM, Zhijiang(wangzhijiang999) < > > [hidden email]> wrote: > > > > > > Thanks @Piotr Nowojski and @Nico Kruber for the good job! > > > > > > I already benefit from this benchmark in the previous PRs. Wish the > > visualization tool becoming stronger to benefit more for the community! > > > > > > Best, > > > Zhijiang > > > ------------------------------------------------------------------ > > > 发件人:Piotr Nowojski <[hidden email]> > > > 发送时间:2018年9月21日(星期五) 22:59 > > > 收件人:dev <[hidden email]> > > > 抄 送:Nico Kruber <[hidden email]> > > > 主 题:Codespeed deployment for Flink > > > > > > Hello community, > > > > > > For almost a year in data Artisans Nico and I were maintaining a setup > > > that continuously evaluates Flink with benchmarks defined at > > > https://github.com/dataArtisans/flink-benchmarks < > > https://github.com/dataArtisans/flink-benchmarks>. With growing interest > > > and after proving useful a couple of times, we have finally decided to > > > publish the web UI layer of this setup. Currently it is accessible via > > > the following (maybe not so?) temporarily url: > > > > > > http://codespeed.dak8s.net:8000 <http://codespeed.dak8s.net:8000/> > > > > > > This is a simple web UI to present performance changes over past and > > > present commits to Apache Flink. It only has a couple of views and the > > > most useful ones are: > > > > > > 1. Timeline > > > 2. Comparison (I recommend to use normalization) > > > > > > Timeline is useful for spotting unintended regressions or unexpected > > > improvements. It is being updated every six hours. > > > Comparison is useful for comparing a given branch (for example a > pending > > > PR) with the master branch. More about that later. > > > > > > The codespeed project on it’s own is just a presentation layer. As > > > mentioned before, the only currently available benchmarks are defined > in > > > the flink-benchmarks repository and they are executed periodically or > on > > > demand by Jenkins on a single bare metal machine. The current setup > > > limits us only to micro benchmarks (they are easier to > > > setup/develop/maintain and have a quicker feedback loop compared to > > > cluster benchmarks) but there is no reason preventing us from setting > up > > > other kinds of benchmarks and upload their results to our codespeed > > > instance as well. > > > > > > Regarding the comparison view. Currently data Artisans’ Flink mirror > > > repository at https://github.com/dataArtisans/flink < > > https://github.com/dataArtisans/flink> is configured to > > > trigger benchmark runs on every commit/change that happens on the > > > benchmark-request branch (We chose to use dataArtisans' repository here > > > because we needed a custom GitHub hook that we couldn’t add to the > > > apache/flink repository). Benchmarking usually takes between one and > two > > > hours. One obvious limitation at the moment is that there is only one > > > comparison view, with one comparison branch, so trying to compare two > > > PRs at the same time is impossible. However we can tackle > > > this problem once it will become a real issue, not only a theoretical > > one. > > > > > > Piotrek & Nico > > > > > > > > |
Free forum by Nabble | Edit this page |