[Proposal] Create a separate sub module for benchmark test

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] Create a separate sub module for benchmark test

Li, Chengxiang
Hi, folks
During work on Flink, I found several micro benchmarks which come from different modules, these benchmarks measure on manual, annotated with Junit annotations, so they got executed during unit test as well. There are some shortage on current implementation:

1.       Benchmark test performance instead of feature, and normally, it takes much more time than unit test. Mixed benchmark with unit test would expand the CI check time.

2.       With mixed with other tests, no warm up, no standalone process... these benchmarks result may not very accurate.
Although looks easy, there are actually many pitfalls about benchmark, so I suggest we create a new sub module for all benchmark test, and import JMH(http://openjdk.java.net/projects/code-tools/jmh/) as the benchmark framework. With the help of JMH, we should get:

1.       More available metrics.

2.       More accurate result.

3.       Focus on benchmark logic only, no need to worry about measure logic.

Thanks
Chengxiang
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] Create a separate sub module for benchmark test

Stephan Ewen
Sounds like a nice idea!

Do you want to make this a new maven project as part of the Flink
repository, or create a dedicated repository for that?

BTW: We are currently not mixing microbenchmarks with test execution. The
code for these benchmarks resides in the test scope of the projects (so it
is not packaged), but it is not executed as part of the UnitTests or
IntegrationTests.

Greetings,
Stephan


On Tue, Sep 22, 2015 at 12:22 PM, Li, Chengxiang <[hidden email]>
wrote:

> Hi, folks
> During work on Flink, I found several micro benchmarks which come from
> different modules, these benchmarks measure on manual, annotated with Junit
> annotations, so they got executed during unit test as well. There are some
> shortage on current implementation:
>
> 1.       Benchmark test performance instead of feature, and normally, it
> takes much more time than unit test. Mixed benchmark with unit test would
> expand the CI check time.
>
> 2.       With mixed with other tests, no warm up, no standalone process...
> these benchmarks result may not very accurate.
> Although looks easy, there are actually many pitfalls about benchmark, so
> I suggest we create a new sub module for all benchmark test, and import JMH(
> http://openjdk.java.net/projects/code-tools/jmh/) as the benchmark
> framework. With the help of JMH, we should get:
>
> 1.       More available metrics.
>
> 2.       More accurate result.
>
> 3.       Focus on benchmark logic only, no need to worry about measure
> logic.
>
> Thanks
> Chengxiang
>
Reply | Threaded
Open this post in threaded view
|

RE: [Proposal] Create a separate sub module for benchmark test

Li, Chengxiang
Thanks for the correctness, Stephan.
For the micro benchmark, I think it should be better to make a new maven project inside Flink.

Thanks
Chengxiang

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Stephan Ewen
Sent: Tuesday, September 22, 2015 8:14 PM
To: [hidden email]
Subject: Re: [Proposal] Create a separate sub module for benchmark test

Sounds like a nice idea!

Do you want to make this a new maven project as part of the Flink repository, or create a dedicated repository for that?

BTW: We are currently not mixing microbenchmarks with test execution. The code for these benchmarks resides in the test scope of the projects (so it is not packaged), but it is not executed as part of the UnitTests or IntegrationTests.

Greetings,
Stephan


On Tue, Sep 22, 2015 at 12:22 PM, Li, Chengxiang <[hidden email]>
wrote:

> Hi, folks
> During work on Flink, I found several micro benchmarks which come from
> different modules, these benchmarks measure on manual, annotated with
> Junit annotations, so they got executed during unit test as well.
> There are some shortage on current implementation:
>
> 1.       Benchmark test performance instead of feature, and normally, it
> takes much more time than unit test. Mixed benchmark with unit test
> would expand the CI check time.
>
> 2.       With mixed with other tests, no warm up, no standalone process...
> these benchmarks result may not very accurate.
> Although looks easy, there are actually many pitfalls about benchmark,
> so I suggest we create a new sub module for all benchmark test, and
> import JMH(
> http://openjdk.java.net/projects/code-tools/jmh/) as the benchmark
> framework. With the help of JMH, we should get:
>
> 1.       More available metrics.
>
> 2.       More accurate result.
>
> 3.       Focus on benchmark logic only, no need to worry about measure
> logic.
>
> Thanks
> Chengxiang
>