[REMINDER] Please only merge pull requests when CI is green and help keep our CI stable

[Robert Metzger]

Hi folks,

As we are approaching the final stages of the Flink 1.12 release cycle, I
would like to remind everybody to *only merge pull requests when the CI
system gives green light*!
I know that the CI system sometimes shows "FAILED" due to CI system
instabilities, but please carefully check those cases to make sure it is
really the CI systems fault, and not an unstable or failing test.
It is much easier to identify and fix an unstable or failing test in the
context of a pull request then after it's been merged.

An essential part of keeping our CI system useful is helping to address
build instabilities early and proactively.
In this phase, I'll try to proactively revert commits introducing severe
issues, or disable failing/unstable tests early to keep the CI system
meaningful.

Please post in this thread or reach out to me personally if you are
uncertain about a build failure or the overall state of the CI system.

Best,
Robert

[Till Rohrmann]

Thanks for the reminder Robert. I think you are completely right. Since we
are getting closer to the feature freeze and therefore also the testing
period, having green builds and only merging PRs where we are sure that CI
passes will decrease the likelihood of introducing new bugs and, hence, it
will decrease the time we need for hardening the release. Of course, this
should also be the modus operandi for not only shortly before the feature
freeze.

Cheers,
Till

On Mon, Oct 19, 2020 at 11:00 AM Robert Metzger <[hidden email]> wrote:

> Hi folks,
>
> As we are approaching the final stages of the Flink 1.12 release cycle, I
> would like to remind everybody to *only merge pull requests when the CI
> system gives green light*!
> I know that the CI system sometimes shows "FAILED" due to CI system
> instabilities, but please carefully check those cases to make sure it is
> really the CI systems fault, and not an unstable or failing test.
> It is much easier to identify and fix an unstable or failing test in the
> context of a pull request then after it's been merged.
>
> An essential part of keeping our CI system useful is helping to address
> build instabilities early and proactively.
> In this phase, I'll try to proactively revert commits introducing severe
> issues, or disable failing/unstable tests early to keep the CI system
> meaningful.
>
> Please post in this thread or reach out to me personally if you are
> uncertain about a build failure or the overall state of the CI system.
>
> Best,
> Robert
>

[Robert Metzger]

Hi all,

Quick update on our CI system during the "hot phase" of the release:
1. We are (for the first time in this release cycle) *hitting our e2e test
processing capacity limit:* All our 20 slots are in use, 13 builds are in
the queue. The other tests are running without delays. I will monitor this
closely.
Please try to use your personal Azure accounts as much as possible for
testing, and only push to your PR once you know the change should pass.

2. I probably made a *mistake in our e2e test scripts, causing broken e2e
tests to not fail the build*.
I'm tracking the issue here:
https://issues.apache.org/jira/browse/FLINK-19839 I'm trying to fix this in
the next hours.
You can see if the e2e test failed if there's a warning with the cache
upload, as with this example:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8385&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529

I have checked the recent master builds, and the e2e tests on master seem
to be stable. Until I've fixed this, please take a quick look onto the CI
overview page of your PR before merging! Thanks & sorry!



On Mon, Oct 19, 2020 at 11:15 AM Till Rohrmann <[hidden email]> wrote:

> Thanks for the reminder Robert. I think you are completely right. Since we
> are getting closer to the feature freeze and therefore also the testing
> period, having green builds and only merging PRs where we are sure that CI
> passes will decrease the likelihood of introducing new bugs and, hence, it
> will decrease the time we need for hardening the release. Of course, this
> should also be the modus operandi for not only shortly before the feature
> freeze.
>
> Cheers,
> Till
>
> On Mon, Oct 19, 2020 at 11:00 AM Robert Metzger <[hidden email]>
> wrote:
>
> > Hi folks,
> >
> > As we are approaching the final stages of the Flink 1.12 release cycle, I
> > would like to remind everybody to *only merge pull requests when the CI
> > system gives green light*!
> > I know that the CI system sometimes shows "FAILED" due to CI system
> > instabilities, but please carefully check those cases to make sure it is
> > really the CI systems fault, and not an unstable or failing test.
> > It is much easier to identify and fix an unstable or failing test in the
> > context of a pull request then after it's been merged.
> >
> > An essential part of keeping our CI system useful is helping to address
> > build instabilities early and proactively.
> > In this phase, I'll try to proactively revert commits introducing severe
> > issues, or disable failing/unstable tests early to keep the CI system
> > meaningful.
> >
> > Please post in this thread or reach out to me personally if you are
> > uncertain about a build failure or the overall state of the CI system.
> >
> > Best,
> > Robert
> >
>

[Robert Metzger]

Quick update: The issue has been resolved.

It was actually not as bad as I thought, because flinkbot was showing the
build as "FAILED", so people probably saw that something is wrong.



On Tue, Oct 27, 2020 at 3:51 PM Robert Metzger <[hidden email]> wrote:

> Hi all,
>
> Quick update on our CI system during the "hot phase" of the release:
> 1. We are (for the first time in this release cycle) *hitting our e2e
> test processing capacity limit:* All our 20 slots are in use, 13 builds
> are in the queue. The other tests are running without delays. I will
> monitor this closely.
> Please try to use your personal Azure accounts as much as possible for
> testing, and only push to your PR once you know the change should pass.
>
> 2. I probably made a *mistake in our e2e test scripts, causing broken e2e
> tests to not fail the build*.
> I'm tracking the issue here:
> https://issues.apache.org/jira/browse/FLINK-19839 I'm trying to fix this
> in the next hours.
> You can see if the e2e test failed if there's a warning with the cache
> upload, as with this example:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8385&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=ff888d9b-cd34-53cc-d90f-3e446d355529
>
> I have checked the recent master builds, and the e2e tests on master seem
> to be stable. Until I've fixed this, please take a quick look onto the CI
> overview page of your PR before merging! Thanks & sorry!
>
>
>
> On Mon, Oct 19, 2020 at 11:15 AM Till Rohrmann <[hidden email]>
> wrote:
>
>> Thanks for the reminder Robert. I think you are completely right. Since we
>> are getting closer to the feature freeze and therefore also the testing
>> period, having green builds and only merging PRs where we are sure that CI
>> passes will decrease the likelihood of introducing new bugs and, hence, it
>> will decrease the time we need for hardening the release. Of course, this
>> should also be the modus operandi for not only shortly before the feature
>> freeze.
>>
>> Cheers,
>> Till
>>
>> On Mon, Oct 19, 2020 at 11:00 AM Robert Metzger <[hidden email]>
>> wrote:
>>
>> > Hi folks,
>> >
>> > As we are approaching the final stages of the Flink 1.12 release cycle,
>> I
>> > would like to remind everybody to *only merge pull requests when the CI
>> > system gives green light*!
>> > I know that the CI system sometimes shows "FAILED" due to CI system
>> > instabilities, but please carefully check those cases to make sure it is
>> > really the CI systems fault, and not an unstable or failing test.
>> > It is much easier to identify and fix an unstable or failing test in the
>> > context of a pull request then after it's been merged.
>> >
>> > An essential part of keeping our CI system useful is helping to address
>> > build instabilities early and proactively.
>> > In this phase, I'll try to proactively revert commits introducing severe
>> > issues, or disable failing/unstable tests early to keep the CI system
>> > meaningful.
>> >
>> > Please post in this thread or reach out to me personally if you are
>> > uncertain about a build failure or the overall state of the CI system.
>> >
>> > Best,
>> > Robert
>> >
>>
>