[ANNOUNCE] Randomized tests

[Arvid Heise-4]

Dear devs,

I just merged FLINK-19520, which adds randomization to all tests that use
the MiniCluster(WithResource). The overall goal is to provide a faster and
better coverage for future changes in the network stack in particular in
conjunction with checkpointing.

We implemented it in a way to fulfill these requirements:

   - Randomization should be seeded in a way that both builds on Azure
   pipelines and local builds will result in the same settings to ease
   debugging and ensure reproducibility.
   - Randomized options should be shown in the test log.
   - Execution order of test cases will not influence the randomization.
   - Randomization is hidden, no change on any test is needed.
   - Randomization only happens during local/remote test execution. User
   deployments are not affected.
   - Test developers are able to avoid randomization by explicitly
   providing values for the randomized configs.

To achieve these goals, we use the git commit id of HEAD + test name to
calculate the seed of the randomization. So if you experience a test
failure on AZP and checkout the same branch/commit, you will get the exact
same configuration.

If for some reason, git commit id changed (e.g., you added some bugfix
commit on top and want to test it) or git commit id cannot be extracted,
you can overwrite the seed with the maven property <test.randomization.seed>
(test name will still be used additionally to that value).

Lastly, you can also add your own randomizations. [1] But please use it
conservatively until we have gained enough experience with the setup.

Best,

Arvid

[1]
https://github.com/AHeise/flink/blob/3d69b21f05abca0de64a2e778c292a7d0694b263/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java#L97

[Till Rohrmann]

Thanks for letting the community know Arvid. Can we document this specific
behaviour somewhere easy to find for others? Maybe we could add it to the
Flink wiki or the how to contribute guide.

Cheers,
Till

On Wed, Feb 10, 2021 at 2:44 PM Arvid Heise <[hidden email]> wrote:

> Dear devs,
>
> I just merged FLINK-19520, which adds randomization to all tests that use
> the MiniCluster(WithResource). The overall goal is to provide a faster and
> better coverage for future changes in the network stack in particular in
> conjunction with checkpointing.
>
> We implemented it in a way to fulfill these requirements:
>
>    - Randomization should be seeded in a way that both builds on Azure
>    pipelines and local builds will result in the same settings to ease
>    debugging and ensure reproducibility.
>    - Randomized options should be shown in the test log.
>    - Execution order of test cases will not influence the randomization.
>    - Randomization is hidden, no change on any test is needed.
>    - Randomization only happens during local/remote test execution. User
>    deployments are not affected.
>    - Test developers are able to avoid randomization by explicitly
>    providing values for the randomized configs.
>
> To achieve these goals, we use the git commit id of HEAD + test name to
> calculate the seed of the randomization. So if you experience a test
> failure on AZP and checkout the same branch/commit, you will get the exact
> same configuration.
>
> If for some reason, git commit id changed (e.g., you added some bugfix
> commit on top and want to test it) or git commit id cannot be extracted,
> you can overwrite the seed with the maven property
> <test.randomization.seed>
> (test name will still be used additionally to that value).
>
> Lastly, you can also add your own randomizations. [1] But please use it
> conservatively until we have gained enough experience with the setup.
>
> Best,
>
> Arvid
>
> [1]
>
> https://github.com/AHeise/flink/blob/3d69b21f05abca0de64a2e778c292a7d0694b263/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java#L97
>

[Piotr Nowojski-5]

Thanks for yours and Chesnay's efforts in this topic.

I also think it would be a good idea to just transcribe your email to some
documentation/wiki page, so that it can be easily found in the future.

Piotrek

czw., 11 lut 2021 o 10:36 Till Rohrmann <[hidden email]> napisał(a):

> Thanks for letting the community know Arvid. Can we document this specific
> behaviour somewhere easy to find for others? Maybe we could add it to the
> Flink wiki or the how to contribute guide.
>
> Cheers,
> Till
>
> On Wed, Feb 10, 2021 at 2:44 PM Arvid Heise <[hidden email]> wrote:
>
> > Dear devs,
> >
> > I just merged FLINK-19520, which adds randomization to all tests that use
> > the MiniCluster(WithResource). The overall goal is to provide a faster
> and
> > better coverage for future changes in the network stack in particular in
> > conjunction with checkpointing.
> >
> > We implemented it in a way to fulfill these requirements:
> >
> >    - Randomization should be seeded in a way that both builds on Azure
> >    pipelines and local builds will result in the same settings to ease
> >    debugging and ensure reproducibility.
> >    - Randomized options should be shown in the test log.
> >    - Execution order of test cases will not influence the randomization.
> >    - Randomization is hidden, no change on any test is needed.
> >    - Randomization only happens during local/remote test execution. User
> >    deployments are not affected.
> >    - Test developers are able to avoid randomization by explicitly
> >    providing values for the randomized configs.
> >
> > To achieve these goals, we use the git commit id of HEAD + test name to
> > calculate the seed of the randomization. So if you experience a test
> > failure on AZP and checkout the same branch/commit, you will get the
> exact
> > same configuration.
> >
> > If for some reason, git commit id changed (e.g., you added some bugfix
> > commit on top and want to test it) or git commit id cannot be extracted,
> > you can overwrite the seed with the maven property
> > <test.randomization.seed>
> > (test name will still be used additionally to that value).
> >
> > Lastly, you can also add your own randomizations. [1] But please use it
> > conservatively until we have gained enough experience with the setup.
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> >
> https://github.com/AHeise/flink/blob/3d69b21f05abca0de64a2e778c292a7d0694b263/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/TestStreamEnvironment.java#L97
> >
>