(DEPRECATED) Apache Flink Mailing List archive.

[VOTE] Release 1.8.0, release candidate #4

Classic

List

Threaded

22 messages Options

Aljoscha Krettek-2

[VOTE] Release 1.8.0, release candidate #4

Hi everyone,
Please review and vote on the release candidate 4 for Flink 1.8.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.8.0-rc4" [5],
* website pull request listing the new release [6]
* website pull request adding announcement blog post [7].

The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Aljoscha

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1215
[5] https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
[6] https://github.com/apache/flink-web/pull/180 <https://github.com/apache/flink-web/pull/180>
[7] https://github.com/apache/flink-web/pull/179 <https://github.com/apache/flink-web/pull/179>

P.S. The difference to the previous RCs is small, you can fetch the tags and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the difference in commits. Its fixes for the issues that led to the cancellation of the previous RCs plus smaller fixes. Most verification/testing that was carried out should apply as is to this RC. Any functional verification that you did on previous RCs should therefore easily carry over to this one.

Piotr Nowojski-3

Re: [VOTE] Release 1.8.0, release candidate #4

+1 from my side. Previously spotted performance regression seems to be gone, or mostly gone.

Piotrek

> On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]> wrote:
>
> Hi everyone,
> Please review and vote on the release candidate 4 for Flink 1.8.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.8.0-rc4" [5],
> * website pull request listing the new release [6]
> * website pull request adding announcement blog post [7].
>
> The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Aljoscha
>
> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1215
> [5] https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> [6] https://github.com/apache/flink-web/pull/180 <https://github.com/apache/flink-web/pull/180>
> [7] https://github.com/apache/flink-web/pull/179 <https://github.com/apache/flink-web/pull/179>
>
> P.S. The difference to the previous RCs is small, you can fetch the tags and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the difference in commits. Its fixes for the issues that led to the cancellation of the previous RCs plus smaller fixes. Most verification/testing that was carried out should apply as is to this RC. Any functional verification that you did on previous RCs should therefore easily carry over to this one.

jincheng sun

Re: [VOTE] Release 1.8.0, release candidate #4

Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!

+1 (non-binding)

I checked the functional things as follows(Without performance
verification):

1. Checking Artifacts：

1). Download the release source code - SUCCESS
2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
3). Download the released JAR - SUCCESS
4). Check if checksums and GPG files match the corresponding release
files - SUCCESS.
5). Verify that the source archives do not contain any binaries -
SUCCESS.
6). Build the source with `mvn clean verify -DskipTests` to ensure all
source files have Apache headers - SUCCESS
7). Check that all POM files point to the same version - SUCCESS
8). Read the `README.md` file to ensure there is nothing unexpected -
SUCCESS

2. Testing Larger Setups

Cluster Environment：7 nodes, jm 1024m, tm 4096m
Testing Jobs: WordCount(Batch&Streaming), DataStreamAllroundTestProgram

1). Use local&hdfs file systems for checkpoints - SUCCESS
2). Use hdfs file systems for input/output -SUCCESS
3). Run examples on YARN(with or without session) - SUCCESS
4). Test failover and recovery. - SUCCESS
5). Test incremental&non-incremental checkpoint - SUCCESS
6). Test connector - kafka -SUCCESS

3. Testing Functionality

1). Built-in tests(linux&mac os)
- `mvn cealn verify` (some test timeout error and test case bug see
FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>, all of
them are not the blocker)
- build for scala 2.11（mvn clean install -P scala-2.11 -DskipTests）
- SUCCESS
- Run the scripted nightly end-to-end test - SUCCESS

2). Quickstarts
- Verify that the quickstarts for Scala with the staging repository
in IntelliJ - SUCCESS
- Verify that the quickstarts for Java with the staging repository in
IntelliJ - SUCCESS

3). Simple Starter Experience and Use Cases

- run all examples from IntelliJ IDE - SUCCESS
- Start a local cluster and verify that the processes - SUCCESS
a. Examine the *.out files (should be empty) and the log files
(should contain no exceptions)
b. Test for Linux, MacOS
c. Shutdown and verify there are no exceptions in the log output
(after shutdown)

- Verify that the examples are running from both ./bin/flink and from
the web-based job submission tool(following items) - SUCCESS
a. Start multiple task managers in the local cluster
b. Change the flink-conf.yml to define more than one task slot (2)
c. Run the examples with a parallelism > 1
d. Examine the log output - no error messages should be encountered

4. Review the PR
- [Add 1.8 Release Blog Post] - Just a reminder, updated the release
date to correct date before merging.

Cheers,
Jincheng

Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：

> +1 from my side. Previously spotted performance regression seems to be
> gone, or mostly gone.
>
> Piotrek
>
> > On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]> wrote:
> >
> > Hi everyone,
> > Please review and vote on the release candidate 4 for Flink 1.8.0, as
> follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release and binary convenience releases to
> be deployed to dist.apache.org [2], which are signed with the key with
> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.8.0-rc4" [5],
> > * website pull request listing the new release [6]
> > * website pull request adding announcement blog post [7].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Aljoscha
> >
> > [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1215
> > [5]
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > [6] https://github.com/apache/flink-web/pull/180 <
> https://github.com/apache/flink-web/pull/180>
> > [7] https://github.com/apache/flink-web/pull/179 <
> https://github.com/apache/flink-web/pull/179>
> >
> > P.S. The difference to the previous RCs is small, you can fetch the tags
> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the
> difference in commits. Its fixes for the issues that led to the
> cancellation of the previous RCs plus smaller fixes. Most
> verification/testing that was carried out should apply as is to this RC.
> Any functional verification that you did on previous RCs should therefore
> easily carry over to this one.
>
>

Aljoscha Krettek-2

Re: [VOTE] Release 1.8.0, release candidate #4

Thanks for the testing done so far!

There has been quite some flakiness on Travis lately, see here: https://travis-ci.org/apache/flink/branches <https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant to release in this state. Looking at the tests you can see that all of the end-to-end tests fail because waiting for the dispatcher to come up times out. I also noticed that this usually takes about 5-8 seconds on Travis, so a 10 second timeout might be a bit low. I pushed commits to increase that to 20 secs. Let’s see what will happen.

I’ll keep you posted!
Aljoscha

> On 25. Mar 2019, at 13:13, jincheng sun <[hidden email]> wrote:
>
> Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
>
> +1 (non-binding)
>
> I checked the functional things as follows(Without performance
> verification):
>
> 1. Checking Artifacts：
>
> 1). Download the release source code - SUCCESS
> 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> 3). Download the released JAR - SUCCESS
> 4). Check if checksums and GPG files match the corresponding release
> files - SUCCESS.
> 5). Verify that the source archives do not contain any binaries -
> SUCCESS.
> 6). Build the source with `mvn clean verify -DskipTests` to ensure all
> source files have Apache headers - SUCCESS
> 7). Check that all POM files point to the same version - SUCCESS
> 8). Read the `README.md` file to ensure there is nothing unexpected -
> SUCCESS
>
> 2. Testing Larger Setups
>
> Cluster Environment：7 nodes, jm 1024m, tm 4096m
> Testing Jobs: WordCount(Batch&Streaming), DataStreamAllroundTestProgram
>
> 1). Use local&hdfs file systems for checkpoints - SUCCESS
> 2). Use hdfs file systems for input/output -SUCCESS
> 3). Run examples on YARN(with or without session) - SUCCESS
> 4). Test failover and recovery. - SUCCESS
> 5). Test incremental&non-incremental checkpoint - SUCCESS
> 6). Test connector - kafka -SUCCESS
>
> 3. Testing Functionality
>
> 1). Built-in tests(linux&mac os)
> - `mvn cealn verify` (some test timeout error and test case bug see
> FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>, all of
> them are not the blocker)
> - build for scala 2.11（mvn clean install -P scala-2.11 -DskipTests）
> - SUCCESS
> - Run the scripted nightly end-to-end test - SUCCESS
>
> 2). Quickstarts
> - Verify that the quickstarts for Scala with the staging repository
> in IntelliJ - SUCCESS
> - Verify that the quickstarts for Java with the staging repository in
> IntelliJ - SUCCESS
>
> 3). Simple Starter Experience and Use Cases
>
> - run all examples from IntelliJ IDE - SUCCESS
> - Start a local cluster and verify that the processes - SUCCESS
> a. Examine the *.out files (should be empty) and the log files
> (should contain no exceptions)
> b. Test for Linux, MacOS
> c. Shutdown and verify there are no exceptions in the log output
> (after shutdown)
>
> - Verify that the examples are running from both ./bin/flink and from
> the web-based job submission tool(following items) - SUCCESS
> a. Start multiple task managers in the local cluster
> b. Change the flink-conf.yml to define more than one task slot (2)
> c. Run the examples with a parallelism > 1
> d. Examine the log output - no error messages should be encountered
>
> 4. Review the PR
> - [Add 1.8 Release Blog Post] - Just a reminder, updated the release
> date to correct date before merging.
>
> Cheers,
> Jincheng
>
> Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
>
>> +1 from my side. Previously spotted performance regression seems to be
>> gone, or mostly gone.
>>
>> Piotrek
>>
>>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]> wrote:
>>>
>>> Hi everyone,
>>> Please review and vote on the release candidate 4 for Flink 1.8.0, as
>> follows:
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release and binary convenience releases to
>> be deployed to dist.apache.org [2], which are signed with the key with
>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "release-1.8.0-rc4" [5],
>>> * website pull request listing the new release [6]
>>> * website pull request adding announcement blog post [7].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> Aljoscha
>>>
>>> [1]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
>>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>>> [4]
>> https://repository.apache.org/content/repositories/orgapacheflink-1215
>>> [5]
>> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
>>> [6] https://github.com/apache/flink-web/pull/180 <
>> https://github.com/apache/flink-web/pull/180>
>>> [7] https://github.com/apache/flink-web/pull/179 <
>> https://github.com/apache/flink-web/pull/179>
>>>
>>> P.S. The difference to the previous RCs is small, you can fetch the tags
>> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the
>> difference in commits. Its fixes for the issues that led to the
>> cancellation of the previous RCs plus smaller fixes. Most
>> verification/testing that was carried out should apply as is to this RC.
>> Any functional verification that you did on previous RCs should therefore
>> easily carry over to this one.
>>
>>

jincheng sun

Re: [VOTE] Release 1.8.0, release candidate #4

Hi Aljoscha, I think you are right, increase the timeout config will fix
this issue. this depends on the resource of Travis. I would like share
some phenomenon during my test (not the flink problem) as follows: :-)

During my testing, `mvn clean verify` and `nightly end-to-end test ` both
consume a lot of machine resources (especially memory/network), and the
network bandwidth requirements of `nightly end-to-end test ` are also very
high. In China, need to use VPN acceleration (100~200Kb before
acceleration, 3~4Mb after acceleration), I have encountered: [Avro
Confluent Schema Registry nightly end-to-end test' failed after 18 minutes
and 15 seconds! Test exited with exit Code 1] takes more than 18 minutes,
the download failed because the network bandwidth is not enough. and it
runs smoothly when using VPN acceleration. The overall end-to-end run was
passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G, Swap:
3.5G). See detail log here
<https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing>
.

Just now, I had checked the Travis for your last commit (Increase startup
timeout in end-to-end tests), in addition to the Cleanup phase, other
phases are successful. here
<https://travis-ci.org/apache/flink/builds/511071777>

In order to verify that our speculation is accurate, I can help with 10 and
20 seconds timeout config on my repo verification to see if 100% recurring
timeout problem. It is already running, we are waiting for the result.
10seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235749>
20seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235598>

Best,
Jincheng

Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：

> Thanks for the testing done so far!
>
> There has been quite some flakiness on Travis lately, see here:
> https://travis-ci.org/apache/flink/branches <
> https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant to
> release in this state. Looking at the tests you can see that all of the
> end-to-end tests fail because waiting for the dispatcher to come up times
> out. I also noticed that this usually takes about 5-8 seconds on Travis, so
> a 10 second timeout might be a bit low. I pushed commits to increase that
> to 20 secs. Let’s see what will happen.
>
> I’ll keep you posted!
> Aljoscha
>
> > On 25. Mar 2019, at 13:13, jincheng sun <[hidden email]>
> wrote:
> >
> > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> >
> > +1 (non-binding)
> >
> > I checked the functional things as follows(Without performance
> > verification):
> >
> > 1. Checking Artifacts：
> >
> > 1). Download the release source code - SUCCESS
> > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > 3). Download the released JAR - SUCCESS
> > 4). Check if checksums and GPG files match the corresponding release
> > files - SUCCESS.
> > 5). Verify that the source archives do not contain any binaries -
> > SUCCESS.
> > 6). Build the source with `mvn clean verify -DskipTests` to ensure all
> > source files have Apache headers - SUCCESS
> > 7). Check that all POM files point to the same version - SUCCESS
> > 8). Read the `README.md` file to ensure there is nothing unexpected -
> > SUCCESS
> >
> > 2. Testing Larger Setups
> >
> > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > Testing Jobs: WordCount(Batch&Streaming), DataStreamAllroundTestProgram
> >
> > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > 2). Use hdfs file systems for input/output -SUCCESS
> > 3). Run examples on YARN(with or without session) - SUCCESS
> > 4). Test failover and recovery. - SUCCESS
> > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > 6). Test connector - kafka -SUCCESS
> >
> > 3. Testing Functionality
> >
> > 1). Built-in tests(linux&mac os)
> > - `mvn cealn verify` (some test timeout error and test case bug see
> > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>, all of
> > them are not the blocker)
> > - build for scala 2.11（mvn clean install -P scala-2.11 -DskipTests）
> > - SUCCESS
> > - Run the scripted nightly end-to-end test - SUCCESS
> >
> > 2). Quickstarts
> > - Verify that the quickstarts for Scala with the staging repository
> > in IntelliJ - SUCCESS
> > - Verify that the quickstarts for Java with the staging repository
> in
> > IntelliJ - SUCCESS
> >
> > 3). Simple Starter Experience and Use Cases
> >
> > - run all examples from IntelliJ IDE - SUCCESS
> > - Start a local cluster and verify that the processes - SUCCESS
> > a. Examine the *.out files (should be empty) and the log files
> > (should contain no exceptions)
> > b. Test for Linux, MacOS
> > c. Shutdown and verify there are no exceptions in the log output
> > (after shutdown)
> >
> > - Verify that the examples are running from both ./bin/flink and
> from
> > the web-based job submission tool(following items) - SUCCESS
> > a. Start multiple task managers in the local cluster
> > b. Change the flink-conf.yml to define more than one task slot (2)
> > c. Run the examples with a parallelism > 1
> > d. Examine the log output - no error messages should be
> encountered
> >
> > 4. Review the PR
> > - [Add 1.8 Release Blog Post] - Just a reminder, updated the release
> > date to correct date before merging.
> >
> > Cheers,
> > Jincheng
> >
> > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> >
> >> +1 from my side. Previously spotted performance regression seems to be
> >> gone, or mostly gone.
> >>
> >> Piotrek
> >>
> >>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]>
> wrote:
> >>>
> >>> Hi everyone,
> >>> Please review and vote on the release candidate 4 for Flink 1.8.0, as
> >> follows:
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release and binary convenience releases to
> >> be deployed to dist.apache.org [2], which are signed with the key with
> >> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag "release-1.8.0-rc4" [5],
> >>> * website pull request listing the new release [6]
> >>> * website pull request adding announcement blog post [7].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> Aljoscha
> >>>
> >>> [1]
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >>> [4]
> >> https://repository.apache.org/content/repositories/orgapacheflink-1215
> >>> [5]
> >>
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> >>> [6] https://github.com/apache/flink-web/pull/180 <
> >> https://github.com/apache/flink-web/pull/180>
> >>> [7] https://github.com/apache/flink-web/pull/179 <
> >> https://github.com/apache/flink-web/pull/179>
> >>>
> >>> P.S. The difference to the previous RCs is small, you can fetch the
> tags
> >> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the
> >> difference in commits. Its fixes for the issues that led to the
> >> cancellation of the previous RCs plus smaller fixes. Most
> >> verification/testing that was carried out should apply as is to this RC.
> >> Any functional verification that you did on previous RCs should
> therefore
> >> easily carry over to this one.
> >>
> >>
>
>

Shaoxuan Wang

Re: [VOTE] Release 1.8.0, release candidate #4

+1 (non-binding)

I tested RC4 with the following items:
- Maven Central Repository contains all artifacts
- Built the source with Maven (ensured all source files have Apache
headers), and executed built-in tests via "mvn clean verify"
- Manually executed the tests in IntelliJ IDE
- Verify that the quickstarts for Scala and Java are working with the
staging repository in IntelliJ
- Checked the benchmark results. The perf regression of
tuple-key-by/statebackend/tumblingWindow are gone, but the regression on
serializer still exists.

Regards,
Shaoxuan

On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <[hidden email]>
wrote:

> Hi Aljoscha, I think you are right, increase the timeout config will fix
> this issue. this depends on the resource of Travis. I would like share
> some phenomenon during my test (not the flink problem) as follows: :-)
>
> During my testing, `mvn clean verify` and `nightly end-to-end test ` both
> consume a lot of machine resources (especially memory/network), and the
> network bandwidth requirements of `nightly end-to-end test ` are also very
> high. In China, need to use VPN acceleration (100~200Kb before
> acceleration, 3~4Mb after acceleration), I have encountered: [Avro
> Confluent Schema Registry nightly end-to-end test' failed after 18 minutes
> and 15 seconds! Test exited with exit Code 1] takes more than 18 minutes,
> the download failed because the network bandwidth is not enough. and it
> runs smoothly when using VPN acceleration. The overall end-to-end run was
> passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G, Swap:
> 3.5G). See detail log here
> <
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> >
> .
>
> Just now, I had checked the Travis for your last commit (Increase startup
> timeout in end-to-end tests), in addition to the Cleanup phase, other
> phases are successful. here
> <https://travis-ci.org/apache/flink/builds/511071777>
>
> In order to verify that our speculation is accurate, I can help with 10 and
> 20 seconds timeout config on my repo verification to see if 100% recurring
> timeout problem. It is already running, we are waiting for the result.
> 10seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235749>
> 20seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235598>
>
> Best,
> Jincheng
>
> Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
>
> > Thanks for the testing done so far!
> >
> > There has been quite some flakiness on Travis lately, see here:
> > https://travis-ci.org/apache/flink/branches <
> > https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant to
> > release in this state. Looking at the tests you can see that all of the
> > end-to-end tests fail because waiting for the dispatcher to come up times
> > out. I also noticed that this usually takes about 5-8 seconds on Travis,
> so
> > a 10 second timeout might be a bit low. I pushed commits to increase that
> > to 20 secs. Let’s see what will happen.
> >
> > I’ll keep you posted!
> > Aljoscha
> >
> > > On 25. Mar 2019, at 13:13, jincheng sun <[hidden email]>
> > wrote:
> > >
> > > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> > >
> > > +1 (non-binding)
> > >
> > > I checked the functional things as follows(Without performance
> > > verification):
> > >
> > > 1. Checking Artifacts：
> > >
> > > 1). Download the release source code - SUCCESS
> > > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > > 3). Download the released JAR - SUCCESS
> > > 4). Check if checksums and GPG files match the corresponding release
> > > files - SUCCESS.
> > > 5). Verify that the source archives do not contain any binaries -
> > > SUCCESS.
> > > 6). Build the source with `mvn clean verify -DskipTests` to ensure
> all
> > > source files have Apache headers - SUCCESS
> > > 7). Check that all POM files point to the same version - SUCCESS
> > > 8). Read the `README.md` file to ensure there is nothing unexpected
> -
> > > SUCCESS
> > >
> > > 2. Testing Larger Setups
> > >
> > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > Testing Jobs: WordCount(Batch&Streaming),
> DataStreamAllroundTestProgram
> > >
> > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > 2). Use hdfs file systems for input/output -SUCCESS
> > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > 4). Test failover and recovery. - SUCCESS
> > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > 6). Test connector - kafka -SUCCESS
> > >
> > > 3. Testing Functionality
> > >
> > > 1). Built-in tests(linux&mac os)
> > > - `mvn cealn verify` (some test timeout error and test case bug
> see
> > > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>, all
> of
> > > them are not the blocker)
> > > - build for scala 2.11（mvn clean install -P scala-2.11
> -DskipTests）
> > > - SUCCESS
> > > - Run the scripted nightly end-to-end test - SUCCESS
> > >
> > > 2). Quickstarts
> > > - Verify that the quickstarts for Scala with the staging
> repository
> > > in IntelliJ - SUCCESS
> > > - Verify that the quickstarts for Java with the staging repository
> > in
> > > IntelliJ - SUCCESS
> > >
> > > 3). Simple Starter Experience and Use Cases
> > >
> > > - run all examples from IntelliJ IDE - SUCCESS
> > > - Start a local cluster and verify that the processes - SUCCESS
> > > a. Examine the *.out files (should be empty) and the log files
> > > (should contain no exceptions)
> > > b. Test for Linux, MacOS
> > > c. Shutdown and verify there are no exceptions in the log output
> > > (after shutdown)
> > >
> > > - Verify that the examples are running from both ./bin/flink and
> > from
> > > the web-based job submission tool(following items) - SUCCESS
> > > a. Start multiple task managers in the local cluster
> > > b. Change the flink-conf.yml to define more than one task slot
> (2)
> > > c. Run the examples with a parallelism > 1
> > > d. Examine the log output - no error messages should be
> > encountered
> > >
> > > 4. Review the PR
> > > - [Add 1.8 Release Blog Post] - Just a reminder, updated the
> release
> > > date to correct date before merging.
> > >
> > > Cheers,
> > > Jincheng
> > >
> > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> > >
> > >> +1 from my side. Previously spotted performance regression seems to be
> > >> gone, or mostly gone.
> > >>
> > >> Piotrek
> > >>
> > >>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]>
> > wrote:
> > >>>
> > >>> Hi everyone,
> > >>> Please review and vote on the release candidate 4 for Flink 1.8.0, as
> > >> follows:
> > >>> [ ] +1, Approve the release
> > >>> [ ] -1, Do not approve the release (please provide specific comments)
> > >>>
> > >>>
> > >>> The complete staging area is available for your review, which
> includes:
> > >>> * JIRA release notes [1],
> > >>> * the official Apache source release and binary convenience releases
> to
> > >> be deployed to dist.apache.org [2], which are signed with the key
> with
> > >> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> > >>> * source code tag "release-1.8.0-rc4" [5],
> > >>> * website pull request listing the new release [6]
> > >>> * website pull request adding announcement blog post [7].
> > >>>
> > >>> The vote will be open for at least 72 hours. It is adopted by
> majority
> > >> approval, with at least 3 PMC affirmative votes.
> > >>>
> > >>> Thanks,
> > >>> Aljoscha
> > >>>
> > >>> [1]
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >>> [4]
> > >>
> https://repository.apache.org/content/repositories/orgapacheflink-1215
> > >>> [5]
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > >>> [6] https://github.com/apache/flink-web/pull/180 <
> > >> https://github.com/apache/flink-web/pull/180>
> > >>> [7] https://github.com/apache/flink-web/pull/179 <
> > >> https://github.com/apache/flink-web/pull/179>
> > >>>
> > >>> P.S. The difference to the previous RCs is small, you can fetch the
> > tags
> > >> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the
> > >> difference in commits. Its fixes for the issues that led to the
> > >> cancellation of the previous RCs plus smaller fixes. Most
> > >> verification/testing that was carried out should apply as is to this
> RC.
> > >> Any functional verification that you did on previous RCs should
> > therefore
> > >> easily carry over to this one.
> > >>
> > >>
> >
> >
>

Kurt Young

Re: [VOTE] Release 1.8.0, release candidate #4

+1 (non-binding)

Checked items:
- checked checksums and GPG files
- verified that the source archives do not contains any binaries
- checked that all POM files point to the same version
- build from source successfully

Best,
Kurt

On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <[hidden email]> wrote:

> +1 (non-binding)
>
> I tested RC4 with the following items:
> - Maven Central Repository contains all artifacts
> - Built the source with Maven (ensured all source files have Apache
> headers), and executed built-in tests via "mvn clean verify"
> - Manually executed the tests in IntelliJ IDE
> - Verify that the quickstarts for Scala and Java are working with the
> staging repository in IntelliJ
> - Checked the benchmark results. The perf regression of
> tuple-key-by/statebackend/tumblingWindow are gone, but the regression on
> serializer still exists.
>
> Regards,
> Shaoxuan
>
> On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <[hidden email]>
> wrote:
>
> > Hi Aljoscha, I think you are right, increase the timeout config will fix
> > this issue. this depends on the resource of Travis. I would like share
> > some phenomenon during my test (not the flink problem) as follows: :-)
> >
> > During my testing, `mvn clean verify` and `nightly end-to-end test ` both
> > consume a lot of machine resources (especially memory/network), and the
> > network bandwidth requirements of `nightly end-to-end test ` are also
> very
> > high. In China, need to use VPN acceleration (100~200Kb before
> > acceleration, 3~4Mb after acceleration), I have encountered: [Avro
> > Confluent Schema Registry nightly end-to-end test' failed after 18
> minutes
> > and 15 seconds! Test exited with exit Code 1] takes more than 18 minutes,
> > the download failed because the network bandwidth is not enough. and it
> > runs smoothly when using VPN acceleration. The overall end-to-end run was
> > passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G,
> Swap:
> > 3.5G). See detail log here
> > <
> >
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > >
> > .
> >
> > Just now, I had checked the Travis for your last commit (Increase startup
> > timeout in end-to-end tests), in addition to the Cleanup phase, other
> > phases are successful. here
> > <https://travis-ci.org/apache/flink/builds/511071777>
> >
> > In order to verify that our speculation is accurate, I can help with 10
> and
> > 20 seconds timeout config on my repo verification to see if 100%
> recurring
> > timeout problem. It is already running, we are waiting for the result.
> > 10seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235749>
> > 20seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235598>
> >
> > Best,
> > Jincheng
> >
> > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
> >
> > > Thanks for the testing done so far!
> > >
> > > There has been quite some flakiness on Travis lately, see here:
> > > https://travis-ci.org/apache/flink/branches <
> > > https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant to
> > > release in this state. Looking at the tests you can see that all of the
> > > end-to-end tests fail because waiting for the dispatcher to come up
> times
> > > out. I also noticed that this usually takes about 5-8 seconds on
> Travis,
> > so
> > > a 10 second timeout might be a bit low. I pushed commits to increase
> that
> > > to 20 secs. Let’s see what will happen.
> > >
> > > I’ll keep you posted!
> > > Aljoscha
> > >
> > > > On 25. Mar 2019, at 13:13, jincheng sun <[hidden email]>
> > > wrote:
> > > >
> > > > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> > > >
> > > > +1 (non-binding)
> > > >
> > > > I checked the functional things as follows(Without performance
> > > > verification):
> > > >
> > > > 1. Checking Artifacts：
> > > >
> > > > 1). Download the release source code - SUCCESS
> > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > > > 3). Download the released JAR - SUCCESS
> > > > 4). Check if checksums and GPG files match the corresponding
> release
> > > > files - SUCCESS.
> > > > 5). Verify that the source archives do not contain any binaries -
> > > > SUCCESS.
> > > > 6). Build the source with `mvn clean verify -DskipTests` to ensure
> > all
> > > > source files have Apache headers - SUCCESS
> > > > 7). Check that all POM files point to the same version - SUCCESS
> > > > 8). Read the `README.md` file to ensure there is nothing
> unexpected
> > -
> > > > SUCCESS
> > > >
> > > > 2. Testing Larger Setups
> > > >
> > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > > Testing Jobs: WordCount(Batch&Streaming),
> > DataStreamAllroundTestProgram
> > > >
> > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > > 2). Use hdfs file systems for input/output -SUCCESS
> > > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > > 4). Test failover and recovery. - SUCCESS
> > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > > 6). Test connector - kafka -SUCCESS
> > > >
> > > > 3. Testing Functionality
> > > >
> > > > 1). Built-in tests(linux&mac os)
> > > > - `mvn cealn verify` (some test timeout error and test case bug
> > see
> > > > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>, all
> > of
> > > > them are not the blocker)
> > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > -DskipTests）
> > > > - SUCCESS
> > > > - Run the scripted nightly end-to-end test - SUCCESS
> > > >
> > > > 2). Quickstarts
> > > > - Verify that the quickstarts for Scala with the staging
> > repository
> > > > in IntelliJ - SUCCESS
> > > > - Verify that the quickstarts for Java with the staging
> repository
> > > in
> > > > IntelliJ - SUCCESS
> > > >
> > > > 3). Simple Starter Experience and Use Cases
> > > >
> > > > - run all examples from IntelliJ IDE - SUCCESS
> > > > - Start a local cluster and verify that the processes - SUCCESS
> > > > a. Examine the *.out files (should be empty) and the log files
> > > > (should contain no exceptions)
> > > > b. Test for Linux, MacOS
> > > > c. Shutdown and verify there are no exceptions in the log
> output
> > > > (after shutdown)
> > > >
> > > > - Verify that the examples are running from both ./bin/flink and
> > > from
> > > > the web-based job submission tool(following items) - SUCCESS
> > > > a. Start multiple task managers in the local cluster
> > > > b. Change the flink-conf.yml to define more than one task slot
> > (2)
> > > > c. Run the examples with a parallelism > 1
> > > > d. Examine the log output - no error messages should be
> > > encountered
> > > >
> > > > 4. Review the PR
> > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated the
> > release
> > > > date to correct date before merging.
> > > >
> > > > Cheers,
> > > > Jincheng
> > > >
> > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> > > >
> > > >> +1 from my side. Previously spotted performance regression seems to
> be
> > > >> gone, or mostly gone.
> > > >>
> > > >> Piotrek
> > > >>
> > > >>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]>
> > > wrote:
> > > >>>
> > > >>> Hi everyone,
> > > >>> Please review and vote on the release candidate 4 for Flink 1.8.0,
> as
> > > >> follows:
> > > >>> [ ] +1, Approve the release
> > > >>> [ ] -1, Do not approve the release (please provide specific
> comments)
> > > >>>
> > > >>>
> > > >>> The complete staging area is available for your review, which
> > includes:
> > > >>> * JIRA release notes [1],
> > > >>> * the official Apache source release and binary convenience
> releases
> > to
> > > >> be deployed to dist.apache.org [2], which are signed with the key
> > with
> > > >> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> > > >>> * source code tag "release-1.8.0-rc4" [5],
> > > >>> * website pull request listing the new release [6]
> > > >>> * website pull request adding announcement blog post [7].
> > > >>>
> > > >>> The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > >> approval, with at least 3 PMC affirmative votes.
> > > >>>
> > > >>> Thanks,
> > > >>> Aljoscha
> > > >>>
> > > >>> [1]
> > > >>
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > > >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > >>> [4]
> > > >>
> > https://repository.apache.org/content/repositories/orgapacheflink-1215
> > > >>> [5]
> > > >>
> > >
> >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > > >>> [6] https://github.com/apache/flink-web/pull/180 <
> > > >> https://github.com/apache/flink-web/pull/180>
> > > >>> [7] https://github.com/apache/flink-web/pull/179 <
> > > >> https://github.com/apache/flink-web/pull/179>
> > > >>>
> > > >>> P.S. The difference to the previous RCs is small, you can fetch the
> > > tags
> > > >> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the
> > > >> difference in commits. Its fixes for the issues that led to the
> > > >> cancellation of the previous RCs plus smaller fixes. Most
> > > >> verification/testing that was carried out should apply as is to this
> > RC.
> > > >> Any functional verification that you did on previous RCs should
> > > therefore
> > > >> easily carry over to this one.
> > > >>
> > > >>
> > >
> > >
> >
>

Yu Li

Re: [VOTE] Release 1.8.0, release candidate #4

+1 (non-binding)

- Checked release notes: OK
- Checked sums and signatures: OK
- Source release
- contains no binaries: OK
- contains no 1.8-SNAPSHOT references: OK
- build from source: OK (8u101)
- mvn clean verify: OK (8u101)
- Binary release
- no examples appear to be missing
- started a cluster; WebUI reachable, example ran successfully
- end-to-end test (all but K8S and docker ones): OK (8u101)
- Repository appears to contain all expected artifacts

Best Regards,
Yu

On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:

> +1 (non-binding)
>
> Checked items:
> - checked checksums and GPG files
> - verified that the source archives do not contains any binaries
> - checked that all POM files point to the same version
> - build from source successfully
>
> Best,
> Kurt
>
>
> On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <[hidden email]>
> wrote:
>
> > +1 (non-binding)
> >
> > I tested RC4 with the following items:
> > - Maven Central Repository contains all artifacts
> > - Built the source with Maven (ensured all source files have Apache
> > headers), and executed built-in tests via "mvn clean verify"
> > - Manually executed the tests in IntelliJ IDE
> > - Verify that the quickstarts for Scala and Java are working with the
> > staging repository in IntelliJ
> > - Checked the benchmark results. The perf regression of
> > tuple-key-by/statebackend/tumblingWindow are gone, but the regression on
> > serializer still exists.
> >
> > Regards,
> > Shaoxuan
> >
> > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <[hidden email]>
> > wrote:
> >
> > > Hi Aljoscha, I think you are right, increase the timeout config will
> fix
> > > this issue. this depends on the resource of Travis. I would like share
> > > some phenomenon during my test (not the flink problem) as follows: :-)
> > >
> > > During my testing, `mvn clean verify` and `nightly end-to-end test `
> both
> > > consume a lot of machine resources (especially memory/network), and the
> > > network bandwidth requirements of `nightly end-to-end test ` are also
> > very
> > > high. In China, need to use VPN acceleration (100~200Kb before
> > > acceleration, 3~4Mb after acceleration), I have encountered: [Avro
> > > Confluent Schema Registry nightly end-to-end test' failed after 18
> > minutes
> > > and 15 seconds! Test exited with exit Code 1] takes more than 18
> minutes,
> > > the download failed because the network bandwidth is not enough. and it
> > > runs smoothly when using VPN acceleration. The overall end-to-end run
> was
> > > passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G,
> > Swap:
> > > 3.5G). See detail log here
> > > <
> > >
> >
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > > >
> > > .
> > >
> > > Just now, I had checked the Travis for your last commit (Increase
> startup
> > > timeout in end-to-end tests), in addition to the Cleanup phase, other
> > > phases are successful. here
> > > <https://travis-ci.org/apache/flink/builds/511071777>
> > >
> > > In order to verify that our speculation is accurate, I can help with 10
> > and
> > > 20 seconds timeout config on my repo verification to see if 100%
> > recurring
> > > timeout problem. It is already running, we are waiting for the result.
> > > 10seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235749
> >
> > > 20seconds <https://travis-ci.org/sunjincheng121/flink/builds/511235598
> >
> > >
> > > Best,
> > > Jincheng
> > >
> > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
> > >
> > > > Thanks for the testing done so far!
> > > >
> > > > There has been quite some flakiness on Travis lately, see here:
> > > > https://travis-ci.org/apache/flink/branches <
> > > > https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant to
> > > > release in this state. Looking at the tests you can see that all of
> the
> > > > end-to-end tests fail because waiting for the dispatcher to come up
> > times
> > > > out. I also noticed that this usually takes about 5-8 seconds on
> > Travis,
> > > so
> > > > a 10 second timeout might be a bit low. I pushed commits to increase
> > that
> > > > to 20 secs. Let’s see what will happen.
> > > >
> > > > I’ll keep you posted!
> > > > Aljoscha
> > > >
> > > > > On 25. Mar 2019, at 13:13, jincheng sun <[hidden email]>
> > > > wrote:
> > > > >
> > > > > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > I checked the functional things as follows(Without performance
> > > > > verification):
> > > > >
> > > > > 1. Checking Artifacts：
> > > > >
> > > > > 1). Download the release source code - SUCCESS
> > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > > > > 3). Download the released JAR - SUCCESS
> > > > > 4). Check if checksums and GPG files match the corresponding
> > release
> > > > > files - SUCCESS.
> > > > > 5). Verify that the source archives do not contain any binaries
> -
> > > > > SUCCESS.
> > > > > 6). Build the source with `mvn clean verify -DskipTests` to
> ensure
> > > all
> > > > > source files have Apache headers - SUCCESS
> > > > > 7). Check that all POM files point to the same version - SUCCESS
> > > > > 8). Read the `README.md` file to ensure there is nothing
> > unexpected
> > > -
> > > > > SUCCESS
> > > > >
> > > > > 2. Testing Larger Setups
> > > > >
> > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > > > Testing Jobs: WordCount(Batch&Streaming),
> > > DataStreamAllroundTestProgram
> > > > >
> > > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > > > 2). Use hdfs file systems for input/output -SUCCESS
> > > > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > > > 4). Test failover and recovery. - SUCCESS
> > > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > > > 6). Test connector - kafka -SUCCESS
> > > > >
> > > > > 3. Testing Functionality
> > > > >
> > > > > 1). Built-in tests(linux&mac os)
> > > > > - `mvn cealn verify` (some test timeout error and test case
> bug
> > > see
> > > > > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>,
> all
> > > of
> > > > > them are not the blocker)
> > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > > -DskipTests）
> > > > > - SUCCESS
> > > > > - Run the scripted nightly end-to-end test - SUCCESS
> > > > >
> > > > > 2). Quickstarts
> > > > > - Verify that the quickstarts for Scala with the staging
> > > repository
> > > > > in IntelliJ - SUCCESS
> > > > > - Verify that the quickstarts for Java with the staging
> > repository
> > > > in
> > > > > IntelliJ - SUCCESS
> > > > >
> > > > > 3). Simple Starter Experience and Use Cases
> > > > >
> > > > > - run all examples from IntelliJ IDE - SUCCESS
> > > > > - Start a local cluster and verify that the processes -
> SUCCESS
> > > > > a. Examine the *.out files (should be empty) and the log
> files
> > > > > (should contain no exceptions)
> > > > > b. Test for Linux, MacOS
> > > > > c. Shutdown and verify there are no exceptions in the log
> > output
> > > > > (after shutdown)
> > > > >
> > > > > - Verify that the examples are running from both ./bin/flink
> and
> > > > from
> > > > > the web-based job submission tool(following items) - SUCCESS
> > > > > a. Start multiple task managers in the local cluster
> > > > > b. Change the flink-conf.yml to define more than one task
> slot
> > > (2)
> > > > > c. Run the examples with a parallelism > 1
> > > > > d. Examine the log output - no error messages should be
> > > > encountered
> > > > >
> > > > > 4. Review the PR
> > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated the
> > > release
> > > > > date to correct date before merging.
> > > > >
> > > > > Cheers,
> > > > > Jincheng
> > > > >
> > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> > > > >
> > > > >> +1 from my side. Previously spotted performance regression seems
> to
> > be
> > > > >> gone, or mostly gone.
> > > > >>
> > > > >> Piotrek
> > > > >>
> > > > >>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <[hidden email]>
> > > > wrote:
> > > > >>>
> > > > >>> Hi everyone,
> > > > >>> Please review and vote on the release candidate 4 for Flink
> 1.8.0,
> > as
> > > > >> follows:
> > > > >>> [ ] +1, Approve the release
> > > > >>> [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > >>>
> > > > >>>
> > > > >>> The complete staging area is available for your review, which
> > > includes:
> > > > >>> * JIRA release notes [1],
> > > > >>> * the official Apache source release and binary convenience
> > releases
> > > to
> > > > >> be deployed to dist.apache.org [2], which are signed with the key
> > > with
> > > > >> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > > > >>> * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > >>> * source code tag "release-1.8.0-rc4" [5],
> > > > >>> * website pull request listing the new release [6]
> > > > >>> * website pull request adding announcement blog post [7].
> > > > >>>
> > > > >>> The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > >> approval, with at least 3 PMC affirmative votes.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Aljoscha
> > > > >>>
> > > > >>> [1]
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > > > >>> [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > >>> [4]
> > > > >>
> > > https://repository.apache.org/content/repositories/orgapacheflink-1215
> > > > >>> [5]
> > > > >>
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > > > >>> [6] https://github.com/apache/flink-web/pull/180 <
> > > > >> https://github.com/apache/flink-web/pull/180>
> > > > >>> [7] https://github.com/apache/flink-web/pull/179 <
> > > > >> https://github.com/apache/flink-web/pull/179>
> > > > >>>
> > > > >>> P.S. The difference to the previous RCs is small, you can fetch
> the
> > > > tags
> > > > >> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see the
> > > > >> difference in commits. Its fixes for the issues that led to the
> > > > >> cancellation of the previous RCs plus smaller fixes. Most
> > > > >> verification/testing that was carried out should apply as is to
> this
> > > RC.
> > > > >> Any functional verification that you did on previous RCs should
> > > > therefore
> > > > >> easily carry over to this one.
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

Xiaowei Jiang

Re: [VOTE] Release 1.8.0, release candidate #4

+1 (non-binding)

- checked checksums and GPG files
- build from source successfully- run end-to-end precommit tests successfully- run end-to-end nightly tests successfully
Xiaowei
On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <[hidden email]> wrote:

+1 (non-binding)

- Checked release notes: OK
- Checked sums and signatures: OK
- Source release
- contains no binaries: OK
- contains no 1.8-SNAPSHOT references: OK
- build from source: OK (8u101)
- mvn clean verify: OK (8u101)
- Binary release
- no examples appear to be missing
- started a cluster; WebUI reachable, example ran successfully
- end-to-end test (all but K8S and docker ones): OK (8u101)
- Repository appears to contain all expected artifacts

Best Regards,
Yu

On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:

vino yang

Re: [VOTE] Release 1.8.0, release candidate #4

+1 (non-binding)

- checked JIRA release note
- ran "mvn package -DskipTests"
- checked signature and checksum
- started a cluster locally and ran some examples in binary
- checked web site announcement's PR

Best,
Vino

Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：

> +1 (non-binding)
>
> - checked checksums and GPG files
> - build from source successfully- run end-to-end precommit tests
> successfully- run end-to-end nightly tests successfully
> Xiaowei
> On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <[hidden email]>
> wrote:
>
> +1 (non-binding)
>
> - Checked release notes: OK
> - Checked sums and signatures: OK
> - Source release
> - contains no binaries: OK
> - contains no 1.8-SNAPSHOT references: OK
> - build from source: OK (8u101)
> - mvn clean verify: OK (8u101)
> - Binary release
> - no examples appear to be missing
> - started a cluster; WebUI reachable, example ran successfully
> - end-to-end test (all but K8S and docker ones): OK (8u101)
> - Repository appears to contain all expected artifacts
>
> Best Regards,
> Yu
>
>
> On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:
>
> > +1 (non-binding)
> >
> > Checked items:
> > - checked checksums and GPG files
> > - verified that the source archives do not contains any binaries
> > - checked that all POM files point to the same version
> > - build from source successfully
> >
> > Best,
> > Kurt
> >
> >
> > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <[hidden email]>
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > I tested RC4 with the following items:
> > > - Maven Central Repository contains all artifacts
> > > - Built the source with Maven (ensured all source files have Apache
> > > headers), and executed built-in tests via "mvn clean verify"
> > > - Manually executed the tests in IntelliJ IDE
> > > - Verify that the quickstarts for Scala and Java are working with the
> > > staging repository in IntelliJ
> > > - Checked the benchmark results. The perf regression of
> > > tuple-key-by/statebackend/tumblingWindow are gone, but the regression
> on
> > > serializer still exists.
> > >
> > > Regards,
> > > Shaoxuan
> > >
> > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi Aljoscha, I think you are right, increase the timeout config will
> > fix
> > > > this issue. this depends on the resource of Travis. I would like
> share
> > > > some phenomenon during my test (not the flink problem) as follows:
> :-)
> > > >
> > > > During my testing, `mvn clean verify` and `nightly end-to-end test `
> > both
> > > > consume a lot of machine resources (especially memory/network), and
> the
> > > > network bandwidth requirements of `nightly end-to-end test ` are also
> > > very
> > > > high. In China, need to use VPN acceleration (100~200Kb before
> > > > acceleration, 3~4Mb after acceleration), I have encountered: [Avro
> > > > Confluent Schema Registry nightly end-to-end test' failed after 18
> > > minutes
> > > > and 15 seconds! Test exited with exit Code 1] takes more than 18
> > minutes,
> > > > the download failed because the network bandwidth is not enough. and
> it
> > > > runs smoothly when using VPN acceleration. The overall end-to-end run
> > was
> > > > passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G,
> > > Swap:
> > > > 3.5G). See detail log here
> > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > > > >
> > > > .
> > > >
> > > > Just now, I had checked the Travis for your last commit (Increase
> > startup
> > > > timeout in end-to-end tests), in addition to the Cleanup phase, other
> > > > phases are successful. here
> > > > <https://travis-ci.org/apache/flink/builds/511071777>
> > > >
> > > > In order to verify that our speculation is accurate, I can help with
> 10
> > > and
> > > > 20 seconds timeout config on my repo verification to see if 100%
> > > recurring
> > > > timeout problem. It is already running, we are waiting for the
> result.
> > > > 10seconds <
> https://travis-ci.org/sunjincheng121/flink/builds/511235749
> > >
> > > > 20seconds <
> https://travis-ci.org/sunjincheng121/flink/builds/511235598
> > >
> > > >
> > > > Best,
> > > > Jincheng
> > > >
> > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
> > > >
> > > > > Thanks for the testing done so far!
> > > > >
> > > > > There has been quite some flakiness on Travis lately, see here:
> > > > > https://travis-ci.org/apache/flink/branches <
> > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant
> to
> > > > > release in this state. Looking at the tests you can see that all of
> > the
> > > > > end-to-end tests fail because waiting for the dispatcher to come up
> > > times
> > > > > out. I also noticed that this usually takes about 5-8 seconds on
> > > Travis,
> > > > so
> > > > > a 10 second timeout might be a bit low. I pushed commits to
> increase
> > > that
> > > > > to 20 secs. Let’s see what will happen.
> > > > >
> > > > > I’ll keep you posted!
> > > > > Aljoscha
> > > > >
> > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > I checked the functional things as follows(Without performance
> > > > > > verification):
> > > > > >
> > > > > > 1. Checking Artifacts：
> > > > > >
> > > > > > 1). Download the release source code - SUCCESS
> > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > > > > > 3). Download the released JAR - SUCCESS
> > > > > > 4). Check if checksums and GPG files match the corresponding
> > > release
> > > > > > files - SUCCESS.
> > > > > > 5). Verify that the source archives do not contain any
> binaries
> > -
> > > > > > SUCCESS.
> > > > > > 6). Build the source with `mvn clean verify -DskipTests` to
> > ensure
> > > > all
> > > > > > source files have Apache headers - SUCCESS
> > > > > > 7). Check that all POM files point to the same version -
> SUCCESS
> > > > > > 8). Read the `README.md` file to ensure there is nothing
> > > unexpected
> > > > -
> > > > > > SUCCESS
> > > > > >
> > > > > > 2. Testing Larger Setups
> > > > > >
> > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > > > > Testing Jobs: WordCount(Batch&Streaming),
> > > > DataStreamAllroundTestProgram
> > > > > >
> > > > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > > > > 2). Use hdfs file systems for input/output -SUCCESS
> > > > > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > > > > 4). Test failover and recovery. - SUCCESS
> > > > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > > > > 6). Test connector - kafka -SUCCESS
> > > > > >
> > > > > > 3. Testing Functionality
> > > > > >
> > > > > > 1). Built-in tests(linux&mac os)
> > > > > > - `mvn cealn verify` (some test timeout error and test case
> > bug
> > > > see
> > > > > > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>,
> > all
> > > > of
> > > > > > them are not the blocker)
> > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > > > -DskipTests）
> > > > > > - SUCCESS
> > > > > > - Run the scripted nightly end-to-end test - SUCCESS
> > > > > >
> > > > > > 2). Quickstarts
> > > > > > - Verify that the quickstarts for Scala with the staging
> > > > repository
> > > > > > in IntelliJ - SUCCESS
> > > > > > - Verify that the quickstarts for Java with the staging
> > > repository
> > > > > in
> > > > > > IntelliJ - SUCCESS
> > > > > >
> > > > > > 3). Simple Starter Experience and Use Cases
> > > > > >
> > > > > > - run all examples from IntelliJ IDE - SUCCESS
> > > > > > - Start a local cluster and verify that the processes -
> > SUCCESS
> > > > > > a. Examine the *.out files (should be empty) and the log
> > files
> > > > > > (should contain no exceptions)
> > > > > > b. Test for Linux, MacOS
> > > > > > c. Shutdown and verify there are no exceptions in the log
> > > output
> > > > > > (after shutdown)
> > > > > >
> > > > > > - Verify that the examples are running from both ./bin/flink
> > and
> > > > > from
> > > > > > the web-based job submission tool(following items) - SUCCESS
> > > > > > a. Start multiple task managers in the local cluster
> > > > > > b. Change the flink-conf.yml to define more than one task
> > slot
> > > > (2)
> > > > > > c. Run the examples with a parallelism > 1
> > > > > > d. Examine the log output - no error messages should be
> > > > > encountered
> > > > > >
> > > > > > 4. Review the PR
> > > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated the
> > > > release
> > > > > > date to correct date before merging.
> > > > > >
> > > > > > Cheers,
> > > > > > Jincheng
> > > > > >
> > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> > > > > >
> > > > > >> +1 from my side. Previously spotted performance regression seems
> > to
> > > be
> > > > > >> gone, or mostly gone.
> > > > > >>
> > > > > >> Piotrek
> > > > > >>
> > > > > >>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> [hidden email]>
> > > > > wrote:
> > > > > >>>
> > > > > >>> Hi everyone,
> > > > > >>> Please review and vote on the release candidate 4 for Flink
> > 1.8.0,
> > > as
> > > > > >> follows:
> > > > > >>> [ ] +1, Approve the release
> > > > > >>> [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > >>>
> > > > > >>>
> > > > > >>> The complete staging area is available for your review, which
> > > > includes:
> > > > > >>> * JIRA release notes [1],
> > > > > >>> * the official Apache source release and binary convenience
> > > releases
> > > > to
> > > > > >> be deployed to dist.apache.org [2], which are signed with the
> key
> > > > with
> > > > > >> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > > > > >>> * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > >>> * source code tag "release-1.8.0-rc4" [5],
> > > > > >>> * website pull request listing the new release [6]
> > > > > >>> * website pull request adding announcement blog post [7].
> > > > > >>>
> > > > > >>> The vote will be open for at least 72 hours. It is adopted by
> > > > majority
> > > > > >> approval, with at least 3 PMC affirmative votes.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Aljoscha
> > > > > >>>
> > > > > >>> [1]
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > > > > >>> [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > > > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > >>> [4]
> > > > > >>
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1215
> > > > > >>> [5]
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > > > > >>> [6] https://github.com/apache/flink-web/pull/180 <
> > > > > >> https://github.com/apache/flink-web/pull/180>
> > > > > >>> [7] https://github.com/apache/flink-web/pull/179 <
> > > > > >> https://github.com/apache/flink-web/pull/179>
> > > > > >>>
> > > > > >>> P.S. The difference to the previous RCs is small, you can fetch
> > the
> > > > > tags
> > > > > >> and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see
> the
> > > > > >> difference in commits. Its fixes for the issues that led to the
> > > > > >> cancellation of the previous RCs plus smaller fixes. Most
> > > > > >> verification/testing that was carried out should apply as is to
> > this
> > > > RC.
> > > > > >> Any functional verification that you did on previous RCs should
> > > > > therefore
> > > > > >> easily carry over to this one.
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >

Congxian Qiu

Re: [VOTE] Release 1.8.0, release candidate #4

+1 （non-binding)

• checked signature and checksum ok
• mvn clean package -DskipTests ok
• Run job on yarn ok
• Test state migration with POJO type (both heap and rocksdb) ok
• - 1.6 -> 1.8
• - 1.7 -> 1.8
• - 1.8 -> 1.8

Best, Congxian
On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>, wrote:

> +1 (non-binding)
>
> - checked JIRA release note
> - ran "mvn package -DskipTests"
> - checked signature and checksum
> - started a cluster locally and ran some examples in binary
> - checked web site announcement's PR
>
> Best,
> Vino
>
>
> Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
>
> > +1 (non-binding)
> >
> > - checked checksums and GPG files
> > - build from source successfully- run end-to-end precommit tests
> > successfully- run end-to-end nightly tests successfully
> > Xiaowei
> > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <[hidden email]>
> > wrote:
> >
> > +1 (non-binding)
> >
> > - Checked release notes: OK
> > - Checked sums and signatures: OK
> > - Source release
> > - contains no binaries: OK
> > - contains no 1.8-SNAPSHOT references: OK
> > - build from source: OK (8u101)
> > - mvn clean verify: OK (8u101)
> > - Binary release
> > - no examples appear to be missing
> > - started a cluster; WebUI reachable, example ran successfully
> > - end-to-end test (all but K8S and docker ones): OK (8u101)
> > - Repository appears to contain all expected artifacts
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:
> >
> > > +1 (non-binding)
> > >
> > > Checked items:
> > > - checked checksums and GPG files
> > > - verified that the source archives do not contains any binaries
> > > - checked that all POM files point to the same version
> > > - build from source successfully
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <[hidden email]>
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > I tested RC4 with the following items:
> > > > - Maven Central Repository contains all artifacts
> > > > - Built the source with Maven (ensured all source files have Apache
> > > > headers), and executed built-in tests via "mvn clean verify"
> > > > - Manually executed the tests in IntelliJ IDE
> > > > - Verify that the quickstarts for Scala and Java are working with the
> > > > staging repository in IntelliJ
> > > > - Checked the benchmark results. The perf regression of
> > > > tuple-key-by/statebackend/tumblingWindow are gone, but the regression
> > on
> > > > serializer still exists.
> > > >
> > > > Regards,
> > > > Shaoxuan
> > > >
> > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <[hidden email]
> > >
> > > > wrote:
> > > >
> > > > > Hi Aljoscha, I think you are right, increase the timeout config will
> > > fix
> > > > > this issue. this depends on the resource of Travis. I would like
> > share
> > > > > some phenomenon during my test (not the flink problem) as follows:
> > :-)
> > > > >
> > > > > During my testing, `mvn clean verify` and `nightly end-to-end test `
> > > both
> > > > > consume a lot of machine resources (especially memory/network), and
> > the
> > > > > network bandwidth requirements of `nightly end-to-end test ` are also
> > > > very
> > > > > high. In China, need to use VPN acceleration (100~200Kb before
> > > > > acceleration, 3~4Mb after acceleration), I have encountered: [Avro
> > > > > Confluent Schema Registry nightly end-to-end test' failed after 18
> > > > minutes
> > > > > and 15 seconds! Test exited with exit Code 1] takes more than 18
> > > minutes,
> > > > > the download failed because the network bandwidth is not enough. and
> > it
> > > > > runs smoothly when using VPN acceleration. The overall end-to-end run
> > > was
> > > > > passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G,
> > > > Swap:
> > > > > 3.5G). See detail log here
> > > > > <
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > > > > >
> > > > > .
> > > > >
> > > > > Just now, I had checked the Travis for your last commit (Increase
> > > startup
> > > > > timeout in end-to-end tests), in addition to the Cleanup phase, other
> > > > > phases are successful. here
> > > > > <https://travis-ci.org/apache/flink/builds/511071777>
> > > > >
> > > > > In order to verify that our speculation is accurate, I can help with
> > 10
> > > > and
> > > > > 20 seconds timeout config on my repo verification to see if 100%
> > > > recurring
> > > > > timeout problem. It is already running, we are waiting for the
> > result.
> > > > > 10seconds <
> > https://travis-ci.org/sunjincheng121/flink/builds/511235749
> > > >
> > > > > 20seconds <
> > https://travis-ci.org/sunjincheng121/flink/builds/511235598
> > > >
> > > > >
> > > > > Best,
> > > > > Jincheng
> > > > >
> > > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
> > > > >
> > > > > > Thanks for the testing done so far!
> > > > > >
> > > > > > There has been quite some flakiness on Travis lately, see here:
> > > > > > https://travis-ci.org/apache/flink/branches <
> > > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant
> > to
> > > > > > release in this state. Looking at the tests you can see that all of
> > > the
> > > > > > end-to-end tests fail because waiting for the dispatcher to come up
> > > > times
> > > > > > out. I also noticed that this usually takes about 5-8 seconds on
> > > > Travis,
> > > > > so
> > > > > > a 10 second timeout might be a bit low. I pushed commits to
> > increase
> > > > that
> > > > > > to 20 secs. Let’s see what will happen.
> > > > > >
> > > > > > I’ll keep you posted!
> > > > > > Aljoscha
> > > > > >
> > > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
> > [hidden email]>
> > > > > > wrote:
> > > > > > >
> > > > > > > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > I checked the functional things as follows(Without performance
> > > > > > > verification):
> > > > > > >
> > > > > > > 1. Checking Artifacts：
> > > > > > >
> > > > > > > 1). Download the release source code - SUCCESS
> > > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > > > > > > 3). Download the released JAR - SUCCESS
> > > > > > > 4). Check if checksums and GPG files match the corresponding
> > > > release
> > > > > > > files - SUCCESS.
> > > > > > > 5). Verify that the source archives do not contain any
> > binaries
> > > -
> > > > > > > SUCCESS.
> > > > > > > 6). Build the source with `mvn clean verify -DskipTests` to
> > > ensure
> > > > > all
> > > > > > > source files have Apache headers - SUCCESS
> > > > > > > 7). Check that all POM files point to the same version -
> > SUCCESS
> > > > > > > 8). Read the `README.md` file to ensure there is nothing
> > > > unexpected
> > > > > -
> > > > > > > SUCCESS
> > > > > > >
> > > > > > > 2. Testing Larger Setups
> > > > > > >
> > > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > > > > > Testing Jobs: WordCount(Batch&Streaming),
> > > > > DataStreamAllroundTestProgram
> > > > > > >
> > > > > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > > > > > 2). Use hdfs file systems for input/output -SUCCESS
> > > > > > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > > > > > 4). Test failover and recovery. - SUCCESS
> > > > > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > > > > > 6). Test connector - kafka -SUCCESS
> > > > > > >
> > > > > > > 3. Testing Functionality
> > > > > > >
> > > > > > > 1). Built-in tests(linux&mac os)
> > > > > > > - `mvn cealn verify` (some test timeout error and test case
> > > bug
> > > > > see
> > > > > > > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>,
> > > all
> > > > > of
> > > > > > > them are not the blocker)
> > > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > > > > -DskipTests）
> > > > > > > - SUCCESS
> > > > > > > - Run the scripted nightly end-to-end test - SUCCESS
> > > > > > >
> > > > > > > 2). Quickstarts
> > > > > > > - Verify that the quickstarts for Scala with the staging
> > > > > repository
> > > > > > > in IntelliJ - SUCCESS
> > > > > > > - Verify that the quickstarts for Java with the staging
> > > > repository
> > > > > > in
> > > > > > > IntelliJ - SUCCESS
> > > > > > >
> > > > > > > 3). Simple Starter Experience and Use Cases
> > > > > > >
> > > > > > > - run all examples from IntelliJ IDE - SUCCESS
> > > > > > > - Start a local cluster and verify that the processes -
> > > SUCCESS
> > > > > > > a. Examine the *.out files (should be empty) and the log
> > > files
> > > > > > > (should contain no exceptions)
> > > > > > > b. Test for Linux, MacOS
> > > > > > > c. Shutdown and verify there are no exceptions in the log
> > > > output
> > > > > > > (after shutdown)
> > > > > > >
> > > > > > > - Verify that the examples are running from both ./bin/flink
> > > and
> > > > > > from
> > > > > > > the web-based job submission tool(following items) - SUCCESS
> > > > > > > a. Start multiple task managers in the local cluster
> > > > > > > b. Change the flink-conf.yml to define more than one task
> > > slot
> > > > > (2)
> > > > > > > c. Run the examples with a parallelism > 1
> > > > > > > d. Examine the log output - no error messages should be
> > > > > > encountered
> > > > > > >
> > > > > > > 4. Review the PR
> > > > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated the
> > > > > release
> > > > > > > date to correct date before merging.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jincheng
> > > > > > >
> > > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> > > > > > >
> > > > > > > > +1 from my side. Previously spotted performance regression seems
> > > to
> > > > be
> > > > > > > > gone, or mostly gone.
> > > > > > > >
> > > > > > > > Piotrek
> > > > > > > >
> > > > > > > > > On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > > Please review and vote on the release candidate 4 for Flink
> > > 1.8.0,
> > > > as
> > > > > > > > follows:
> > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > comments)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The complete staging area is available for your review, which
> > > > > includes:
> > > > > > > > > * JIRA release notes [1],
> > > > > > > > > * the official Apache source release and binary convenience
> > > > releases
> > > > > to
> > > > > > > > be deployed to dist.apache.org [2], which are signed with the
> > key
> > > > > with
> > > > > > > > fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > > > > > > > > * all artifacts to be deployed to the Maven Central Repository
> > > [4],
> > > > > > > > > * source code tag "release-1.8.0-rc4" [5],
> > > > > > > > > * website pull request listing the new release [6]
> > > > > > > > > * website pull request adding announcement blog post [7].
> > > > > > > > >
> > > > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > > majority
> > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Aljoscha
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > > > > > > > > [2]
> > > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > > > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > > > [4]
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1215
> > > > > > > > > [5]
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > > > > > > > > [6] https://github.com/apache/flink-web/pull/180 <
> > > > > > > > https://github.com/apache/flink-web/pull/180>
> > > > > > > > > [7] https://github.com/apache/flink-web/pull/179 <
> > > > > > > > https://github.com/apache/flink-web/pull/179>
> > > > > > > > >
> > > > > > > > > P.S. The difference to the previous RCs is small, you can fetch
> > > the
> > > > > > tags
> > > > > > > > and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see
> > the
> > > > > > > > difference in commits. Its fixes for the issues that led to the
> > > > > > > > cancellation of the previous RCs plus smaller fixes. Most
> > > > > > > > verification/testing that was carried out should apply as is to
> > > this
> > > > > RC.
> > > > > > > > Any functional verification that you did on previous RCs should
> > > > > > therefore
> > > > > > > > easily carry over to this one.
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >

Aljoscha Krettek-3

Re: [VOTE] Release 1.8.0, release candidate #4

By now, I'm reasonably sure that the test instabilities on the end-to-end test are only instabilities. I pushed changes to increase timeouts to make the tests more stable. As in any project, there will always be bugs but I think we could release this RC4 and be reasonably sure that it works well.

Now, we only need to have the required number of PMC votes.

On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:

> +1 （non-binding)
>
> • checked signature and checksum ok
> • mvn clean package -DskipTests ok
> • Run job on yarn ok
> • Test state migration with POJO type (both heap and rocksdb) ok
> • - 1.6 -> 1.8
> • - 1.7 -> 1.8
> • - 1.8 -> 1.8
>
>
> Best, Congxian
> On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>, wrote:
> > +1 (non-binding)
> >
> > - checked JIRA release note
> > - ran "mvn package -DskipTests"
> > - checked signature and checksum
> > - started a cluster locally and ran some examples in binary
> > - checked web site announcement's PR
> >
> > Best,
> > Vino
> >
> >
> > Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
> >
> > > +1 (non-binding)
> > >
> > > - checked checksums and GPG files
> > > - build from source successfully- run end-to-end precommit tests
> > > successfully- run end-to-end nightly tests successfully
> > > Xiaowei
> > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <[hidden email]>
> > > wrote:
> > >
> > > +1 (non-binding)
> > >
> > > - Checked release notes: OK
> > > - Checked sums and signatures: OK
> > > - Source release
> > > - contains no binaries: OK
> > > - contains no 1.8-SNAPSHOT references: OK
> > > - build from source: OK (8u101)
> > > - mvn clean verify: OK (8u101)
> > > - Binary release
> > > - no examples appear to be missing
> > > - started a cluster; WebUI reachable, example ran successfully
> > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> > > - Repository appears to contain all expected artifacts
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Checked items:
> > > > - checked checksums and GPG files
> > > > - verified that the source archives do not contains any binaries
> > > > - checked that all POM files point to the same version
> > > > - build from source successfully
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <[hidden email]>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > I tested RC4 with the following items:
> > > > > - Maven Central Repository contains all artifacts
> > > > > - Built the source with Maven (ensured all source files have Apache
> > > > > headers), and executed built-in tests via "mvn clean verify"
> > > > > - Manually executed the tests in IntelliJ IDE
> > > > > - Verify that the quickstarts for Scala and Java are working with the
> > > > > staging repository in IntelliJ
> > > > > - Checked the benchmark results. The perf regression of
> > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the regression
> > > on
> > > > > serializer still exists.
> > > > >
> > > > > Regards,
> > > > > Shaoxuan
> > > > >
> > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <[hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Aljoscha, I think you are right, increase the timeout config will
> > > > fix
> > > > > > this issue. this depends on the resource of Travis. I would like
> > > share
> > > > > > some phenomenon during my test (not the flink problem) as follows:
> > > :-)
> > > > > >
> > > > > > During my testing, `mvn clean verify` and `nightly end-to-end test `
> > > > both
> > > > > > consume a lot of machine resources (especially memory/network), and
> > > the
> > > > > > network bandwidth requirements of `nightly end-to-end test ` are also
> > > > > very
> > > > > > high. In China, need to use VPN acceleration (100~200Kb before
> > > > > > acceleration, 3~4Mb after acceleration), I have encountered: [Avro
> > > > > > Confluent Schema Registry nightly end-to-end test' failed after 18
> > > > > minutes
> > > > > > and 15 seconds! Test exited with exit Code 1] takes more than 18
> > > > minutes,
> > > > > > the download failed because the network bandwidth is not enough. and
> > > it
> > > > > > runs smoothly when using VPN acceleration. The overall end-to-end run
> > > > was
> > > > > > passed twice. The Docker resource configuration (CUPs 7, Mem: 28.7G,
> > > > > Swap:
> > > > > > 3.5G). See detail log here
> > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > > > > > >
> > > > > > .
> > > > > >
> > > > > > Just now, I had checked the Travis for your last commit (Increase
> > > > startup
> > > > > > timeout in end-to-end tests), in addition to the Cleanup phase, other
> > > > > > phases are successful. here
> > > > > > <https://travis-ci.org/apache/flink/builds/511071777>
> > > > > >
> > > > > > In order to verify that our speculation is accurate, I can help with
> > > 10
> > > > > and
> > > > > > 20 seconds timeout config on my repo verification to see if 100%
> > > > > recurring
> > > > > > timeout problem. It is already running, we are waiting for the
> > > result.
> > > > > > 10seconds <
> > > https://travis-ci.org/sunjincheng121/flink/builds/511235749
> > > > >
> > > > > > 20seconds <
> > > https://travis-ci.org/sunjincheng121/flink/builds/511235598
> > > > >
> > > > > >
> > > > > > Best,
> > > > > > Jincheng
> > > > > >
> > > > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
> > > > > >
> > > > > > > Thanks for the testing done so far!
> > > > > > >
> > > > > > > There has been quite some flakiness on Travis lately, see here:
> > > > > > > https://travis-ci.org/apache/flink/branches <
> > > > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit hesitant
> > > to
> > > > > > > release in this state. Looking at the tests you can see that all of
> > > > the
> > > > > > > end-to-end tests fail because waiting for the dispatcher to come up
> > > > > times
> > > > > > > out. I also noticed that this usually takes about 5-8 seconds on
> > > > > Travis,
> > > > > > so
> > > > > > > a 10 second timeout might be a bit low. I pushed commits to
> > > increase
> > > > > that
> > > > > > > to 20 secs. Let’s see what will happen.
> > > > > > >
> > > > > > > I’ll keep you posted!
> > > > > > > Aljoscha
> > > > > > >
> > > > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Great thanks for preparing the RC4 of Flink 1.8.0, Aljoscha!
> > > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > I checked the functional things as follows(Without performance
> > > > > > > > verification):
> > > > > > > >
> > > > > > > > 1. Checking Artifacts：
> > > > > > > >
> > > > > > > > 1). Download the release source code - SUCCESS
> > > > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 - SUCCESS
> > > > > > > > 3). Download the released JAR - SUCCESS
> > > > > > > > 4). Check if checksums and GPG files match the corresponding
> > > > > release
> > > > > > > > files - SUCCESS.
> > > > > > > > 5). Verify that the source archives do not contain any
> > > binaries
> > > > -
> > > > > > > > SUCCESS.
> > > > > > > > 6). Build the source with `mvn clean verify -DskipTests` to
> > > > ensure
> > > > > > all
> > > > > > > > source files have Apache headers - SUCCESS
> > > > > > > > 7). Check that all POM files point to the same version -
> > > SUCCESS
> > > > > > > > 8). Read the `README.md` file to ensure there is nothing
> > > > > unexpected
> > > > > > -
> > > > > > > > SUCCESS
> > > > > > > >
> > > > > > > > 2. Testing Larger Setups
> > > > > > > >
> > > > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > > > > > > Testing Jobs: WordCount(Batch&Streaming),
> > > > > > DataStreamAllroundTestProgram
> > > > > > > >
> > > > > > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > > > > > > 2). Use hdfs file systems for input/output -SUCCESS
> > > > > > > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > > > > > > 4). Test failover and recovery. - SUCCESS
> > > > > > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > > > > > > 6). Test connector - kafka -SUCCESS
> > > > > > > >
> > > > > > > > 3. Testing Functionality
> > > > > > > >
> > > > > > > > 1). Built-in tests(linux&mac os)
> > > > > > > > - `mvn cealn verify` (some test timeout error and test case
> > > > bug
> > > > > > see
> > > > > > > > FLINK-12001 <https://issues.apache.org/jira/browse/FLINK-12001>,
> > > > all
> > > > > > of
> > > > > > > > them are not the blocker)
> > > > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > > > > > -DskipTests）
> > > > > > > > - SUCCESS
> > > > > > > > - Run the scripted nightly end-to-end test - SUCCESS
> > > > > > > >
> > > > > > > > 2). Quickstarts
> > > > > > > > - Verify that the quickstarts for Scala with the staging
> > > > > > repository
> > > > > > > > in IntelliJ - SUCCESS
> > > > > > > > - Verify that the quickstarts for Java with the staging
> > > > > repository
> > > > > > > in
> > > > > > > > IntelliJ - SUCCESS
> > > > > > > >
> > > > > > > > 3). Simple Starter Experience and Use Cases
> > > > > > > >
> > > > > > > > - run all examples from IntelliJ IDE - SUCCESS
> > > > > > > > - Start a local cluster and verify that the processes -
> > > > SUCCESS
> > > > > > > > a. Examine the *.out files (should be empty) and the log
> > > > files
> > > > > > > > (should contain no exceptions)
> > > > > > > > b. Test for Linux, MacOS
> > > > > > > > c. Shutdown and verify there are no exceptions in the log
> > > > > output
> > > > > > > > (after shutdown)
> > > > > > > >
> > > > > > > > - Verify that the examples are running from both ./bin/flink
> > > > and
> > > > > > > from
> > > > > > > > the web-based job submission tool(following items) - SUCCESS
> > > > > > > > a. Start multiple task managers in the local cluster
> > > > > > > > b. Change the flink-conf.yml to define more than one task
> > > > slot
> > > > > > (2)
> > > > > > > > c. Run the examples with a parallelism > 1
> > > > > > > > d. Examine the log output - no error messages should be
> > > > > > > encountered
> > > > > > > >
> > > > > > > > 4. Review the PR
> > > > > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated the
> > > > > > release
> > > > > > > > date to correct date before merging.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jincheng
> > > > > > > >
> > > > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一下午4:11写道：
> > > > > > > >
> > > > > > > > > +1 from my side. Previously spotted performance regression seems
> > > > to
> > > > > be
> > > > > > > > > gone, or mostly gone.
> > > > > > > > >
> > > > > > > > > Piotrek
> > > > > > > > >
> > > > > > > > > > On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > > Please review and vote on the release candidate 4 for Flink
> > > > 1.8.0,
> > > > > as
> > > > > > > > > follows:
> > > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > > [ ] -1, Do not approve the release (please provide specific
> > > > > comments)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The complete staging area is available for your review, which
> > > > > > includes:
> > > > > > > > > > * JIRA release notes [1],
> > > > > > > > > > * the official Apache source release and binary convenience
> > > > > releases
> > > > > > to
> > > > > > > > > be deployed to dist.apache.org [2], which are signed with the
> > > key
> > > > > > with
> > > > > > > > > fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > > > > > > > > > * all artifacts to be deployed to the Maven Central Repository
> > > > [4],
> > > > > > > > > > * source code tag "release-1.8.0-rc4" [5],
> > > > > > > > > > * website pull request listing the new release [6]
> > > > > > > > > > * website pull request adding announcement blog post [7].
> > > > > > > > > >
> > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > > > > majority
> > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Aljoscha
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > > > > > > > > > [2]
> > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > > > > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > > > > [4]
> > > > > > > > >
> > > > > >
> > > https://repository.apache.org/content/repositories/orgapacheflink-1215
> > > > > > > > > > [5]
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > > > > > > > > > [6] https://github.com/apache/flink-web/pull/180 <
> > > > > > > > > https://github.com/apache/flink-web/pull/180>
> > > > > > > > > > [7] https://github.com/apache/flink-web/pull/179 <
> > > > > > > > > https://github.com/apache/flink-web/pull/179>
> > > > > > > > > >
> > > > > > > > > > P.S. The difference to the previous RCs is small, you can fetch
> > > > the
> > > > > > > tags
> > > > > > > > > and do a "git log release-1.8.0-rc1..release-1.8.0-rc4” to see
> > > the
> > > > > > > > > difference in commits. Its fixes for the issues that led to the
> > > > > > > > > cancellation of the previous RCs plus smaller fixes. Most
> > > > > > > > > verification/testing that was carried out should apply as is to
> > > > this
> > > > > > RC.
> > > > > > > > > Any functional verification that you did on previous RCs should
> > > > > > > therefore
> > > > > > > > > easily carry over to this one.
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>

Tzu-Li (Gordon) Tai

Re: [VOTE] Release 1.8.0, release candidate #4

@Shaoxuan

The drop in the serializerAvro benchmark, as explained earlier in previous
voting threads of earlier RCs, was due to a slower job initialization phase
caused by slower deserialization of the AvroSerializer.
Piotr also pointed out that after the number of records was increased in
the serializer benchmarks, this drop was no longer observable before /
after the changes in mid February.
IMO, this is not critical as it does not affect the per-record performance
/ throughput, and therefore should not block this release.

On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <[hidden email]>
wrote:

> By now, I'm reasonably sure that the test instabilities on the end-to-end
> test are only instabilities. I pushed changes to increase timeouts to make
> the tests more stable. As in any project, there will always be bugs but I
> think we could release this RC4 and be reasonably sure that it works well.
>
> Now, we only need to have the required number of PMC votes.
>
> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> > +1 （non-binding)
> >
> > • checked signature and checksum ok
> > • mvn clean package -DskipTests ok
> > • Run job on yarn ok
> > • Test state migration with POJO type (both heap and rocksdb) ok
> > • - 1.6 -> 1.8
> > • - 1.7 -> 1.8
> > • - 1.8 -> 1.8
> >
> >
> > Best, Congxian
> > On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>, wrote:
> > > +1 (non-binding)
> > >
> > > - checked JIRA release note
> > > - ran "mvn package -DskipTests"
> > > - checked signature and checksum
> > > - started a cluster locally and ran some examples in binary
> > > - checked web site announcement's PR
> > >
> > > Best,
> > > Vino
> > >
> > >
> > > Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > - checked checksums and GPG files
> > > > - build from source successfully- run end-to-end precommit tests
> > > > successfully- run end-to-end nightly tests successfully
> > > > Xiaowei
> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> [hidden email]>
> > > > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > - Checked release notes: OK
> > > > - Checked sums and signatures: OK
> > > > - Source release
> > > > - contains no binaries: OK
> > > > - contains no 1.8-SNAPSHOT references: OK
> > > > - build from source: OK (8u101)
> > > > - mvn clean verify: OK (8u101)
> > > > - Binary release
> > > > - no examples appear to be missing
> > > > - started a cluster; WebUI reachable, example ran successfully
> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> > > > - Repository appears to contain all expected artifacts
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Checked items:
> > > > > - checked checksums and GPG files
> > > > > - verified that the source archives do not contains any binaries
> > > > > - checked that all POM files point to the same version
> > > > > - build from source successfully
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > I tested RC4 with the following items:
> > > > > > - Maven Central Repository contains all artifacts
> > > > > > - Built the source with Maven (ensured all source files have
> Apache
> > > > > > headers), and executed built-in tests via "mvn clean verify"
> > > > > > - Manually executed the tests in IntelliJ IDE
> > > > > > - Verify that the quickstarts for Scala and Java are working
> with the
> > > > > > staging repository in IntelliJ
> > > > > > - Checked the benchmark results. The perf regression of
> > > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the
> regression
> > > > on
> > > > > > serializer still exists.
> > > > > >
> > > > > > Regards,
> > > > > > Shaoxuan
> > > > > >
> > > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
> [hidden email]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Aljoscha, I think you are right, increase the timeout
> config will
> > > > > fix
> > > > > > > this issue. this depends on the resource of Travis. I would
> like
> > > > share
> > > > > > > some phenomenon during my test (not the flink problem) as
> follows:
> > > > :-)
> > > > > > >
> > > > > > > During my testing, `mvn clean verify` and `nightly end-to-end
> test `
> > > > > both
> > > > > > > consume a lot of machine resources (especially
> memory/network), and
> > > > the
> > > > > > > network bandwidth requirements of `nightly end-to-end test `
> are also
> > > > > > very
> > > > > > > high. In China, need to use VPN acceleration (100~200Kb before
> > > > > > > acceleration, 3~4Mb after acceleration), I have encountered:
> [Avro
> > > > > > > Confluent Schema Registry nightly end-to-end test' failed
> after 18
> > > > > > minutes
> > > > > > > and 15 seconds! Test exited with exit Code 1] takes more than
> 18
> > > > > minutes,
> > > > > > > the download failed because the network bandwidth is not
> enough. and
> > > > it
> > > > > > > runs smoothly when using VPN acceleration. The overall
> end-to-end run
> > > > > was
> > > > > > > passed twice. The Docker resource configuration (CUPs 7, Mem:
> 28.7G,
> > > > > > Swap:
> > > > > > > 3.5G). See detail log here
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > > > > > > >
> > > > > > > .
> > > > > > >
> > > > > > > Just now, I had checked the Travis for your last commit
> (Increase
> > > > > startup
> > > > > > > timeout in end-to-end tests), in addition to the Cleanup
> phase, other
> > > > > > > phases are successful. here
> > > > > > > <https://travis-ci.org/apache/flink/builds/511071777>
> > > > > > >
> > > > > > > In order to verify that our speculation is accurate, I can
> help with
> > > > 10
> > > > > > and
> > > > > > > 20 seconds timeout config on my repo verification to see if
> 100%
> > > > > > recurring
> > > > > > > timeout problem. It is already running, we are waiting for the
> > > > result.
> > > > > > > 10seconds <
> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235749
> > > > > >
> > > > > > > 20seconds <
> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235598
> > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Jincheng
> > > > > > >
> > > > > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二上午1:04写道：
> > > > > > >
> > > > > > > > Thanks for the testing done so far!
> > > > > > > >
> > > > > > > > There has been quite some flakiness on Travis lately, see
> here:
> > > > > > > > https://travis-ci.org/apache/flink/branches <
> > > > > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit
> hesitant
> > > > to
> > > > > > > > release in this state. Looking at the tests you can see that
> all of
> > > > > the
> > > > > > > > end-to-end tests fail because waiting for the dispatcher to
> come up
> > > > > > times
> > > > > > > > out. I also noticed that this usually takes about 5-8
> seconds on
> > > > > > Travis,
> > > > > > > so
> > > > > > > > a 10 second timeout might be a bit low. I pushed commits to
> > > > increase
> > > > > > that
> > > > > > > > to 20 secs. Let’s see what will happen.
> > > > > > > >
> > > > > > > > I’ll keep you posted!
> > > > > > > > Aljoscha
> > > > > > > >
> > > > > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Great thanks for preparing the RC4 of Flink 1.8.0,
> Aljoscha!
> > > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > >
> > > > > > > > > I checked the functional things as follows(Without
> performance
> > > > > > > > > verification):
> > > > > > > > >
> > > > > > > > > 1. Checking Artifacts：
> > > > > > > > >
> > > > > > > > > 1). Download the release source code - SUCCESS
> > > > > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 -
> SUCCESS
> > > > > > > > > 3). Download the released JAR - SUCCESS
> > > > > > > > > 4). Check if checksums and GPG files match the
> corresponding
> > > > > > release
> > > > > > > > > files - SUCCESS.
> > > > > > > > > 5). Verify that the source archives do not contain any
> > > > binaries
> > > > > -
> > > > > > > > > SUCCESS.
> > > > > > > > > 6). Build the source with `mvn clean verify -DskipTests` to
> > > > > ensure
> > > > > > > all
> > > > > > > > > source files have Apache headers - SUCCESS
> > > > > > > > > 7). Check that all POM files point to the same version -
> > > > SUCCESS
> > > > > > > > > 8). Read the `README.md` file to ensure there is nothing
> > > > > > unexpected
> > > > > > > -
> > > > > > > > > SUCCESS
> > > > > > > > >
> > > > > > > > > 2. Testing Larger Setups
> > > > > > > > >
> > > > > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > > > > > > > > Testing Jobs: WordCount(Batch&Streaming),
> > > > > > > DataStreamAllroundTestProgram
> > > > > > > > >
> > > > > > > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
> > > > > > > > > 2). Use hdfs file systems for input/output -SUCCESS
> > > > > > > > > 3). Run examples on YARN(with or without session) - SUCCESS
> > > > > > > > > 4). Test failover and recovery. - SUCCESS
> > > > > > > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
> > > > > > > > > 6). Test connector - kafka -SUCCESS
> > > > > > > > >
> > > > > > > > > 3. Testing Functionality
> > > > > > > > >
> > > > > > > > > 1). Built-in tests(linux&mac os)
> > > > > > > > > - `mvn cealn verify` (some test timeout error and test case
> > > > > bug
> > > > > > > see
> > > > > > > > > FLINK-12001 <
> https://issues.apache.org/jira/browse/FLINK-12001>,
> > > > > all
> > > > > > > of
> > > > > > > > > them are not the blocker)
> > > > > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > > > > > > -DskipTests）
> > > > > > > > > - SUCCESS
> > > > > > > > > - Run the scripted nightly end-to-end test - SUCCESS
> > > > > > > > >
> > > > > > > > > 2). Quickstarts
> > > > > > > > > - Verify that the quickstarts for Scala with the staging
> > > > > > > repository
> > > > > > > > > in IntelliJ - SUCCESS
> > > > > > > > > - Verify that the quickstarts for Java with the staging
> > > > > > repository
> > > > > > > > in
> > > > > > > > > IntelliJ - SUCCESS
> > > > > > > > >
> > > > > > > > > 3). Simple Starter Experience and Use Cases
> > > > > > > > >
> > > > > > > > > - run all examples from IntelliJ IDE - SUCCESS
> > > > > > > > > - Start a local cluster and verify that the processes -
> > > > > SUCCESS
> > > > > > > > > a. Examine the *.out files (should be empty) and the log
> > > > > files
> > > > > > > > > (should contain no exceptions)
> > > > > > > > > b. Test for Linux, MacOS
> > > > > > > > > c. Shutdown and verify there are no exceptions in the log
> > > > > > output
> > > > > > > > > (after shutdown)
> > > > > > > > >
> > > > > > > > > - Verify that the examples are running from both
> ./bin/flink
> > > > > and
> > > > > > > > from
> > > > > > > > > the web-based job submission tool(following items) -
> SUCCESS
> > > > > > > > > a. Start multiple task managers in the local cluster
> > > > > > > > > b. Change the flink-conf.yml to define more than one task
> > > > > slot
> > > > > > > (2)
> > > > > > > > > c. Run the examples with a parallelism > 1
> > > > > > > > > d. Examine the log output - no error messages should be
> > > > > > > > encountered
> > > > > > > > >
> > > > > > > > > 4. Review the PR
> > > > > > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated
> the
> > > > > > > release
> > > > > > > > > date to correct date before merging.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Jincheng
> > > > > > > > >
> > > > > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一
> 下午4:11写道：
> > > > > > > > >
> > > > > > > > > > +1 from my side. Previously spotted performance
> regression seems
> > > > > to
> > > > > > be
> > > > > > > > > > gone, or mostly gone.
> > > > > > > > > >
> > > > > > > > > > Piotrek
> > > > > > > > > >
> > > > > > > > > > > On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > > Please review and vote on the release candidate 4 for
> Flink
> > > > > 1.8.0,
> > > > > > as
> > > > > > > > > > follows:
> > > > > > > > > > > [ ] +1, Approve the release
> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> specific
> > > > > > comments)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The complete staging area is available for your
> review, which
> > > > > > > includes:
> > > > > > > > > > > * JIRA release notes [1],
> > > > > > > > > > > * the official Apache source release and binary
> convenience
> > > > > > releases
> > > > > > > to
> > > > > > > > > > be deployed to dist.apache.org [2], which are signed
> with the
> > > > key
> > > > > > > with
> > > > > > > > > > fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3],
> > > > > > > > > > > * all artifacts to be deployed to the Maven Central
> Repository
> > > > > [4],
> > > > > > > > > > > * source code tag "release-1.8.0-rc4" [5],
> > > > > > > > > > > * website pull request listing the new release [6]
> > > > > > > > > > > * website pull request adding announcement blog post
> [7].
> > > > > > > > > > >
> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> adopted by
> > > > > > > majority
> > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Aljoscha
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > > > > > > > > > > [2]
> > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > > > > > > > > > > [3]
> https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > > > > > > [4]
> > > > > > > > > >
> > > > > > >
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1215
> > > > > > > > > > > [5]
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > > > > > > > > > > [6] https://github.com/apache/flink-web/pull/180 <
> > > > > > > > > > https://github.com/apache/flink-web/pull/180>
> > > > > > > > > > > [7] https://github.com/apache/flink-web/pull/179 <
> > > > > > > > > > https://github.com/apache/flink-web/pull/179>
> > > > > > > > > > >
> > > > > > > > > > > P.S. The difference to the previous RCs is small, you
> can fetch
> > > > > the
> > > > > > > > tags
> > > > > > > > > > and do a "git log release-1.8.0-rc1..release-1.8.0-rc4”
> to see
> > > > the
> > > > > > > > > > difference in commits. Its fixes for the issues that led
> to the
> > > > > > > > > > cancellation of the previous RCs plus smaller fixes. Most
> > > > > > > > > > verification/testing that was carried out should apply
> as is to
> > > > > this
> > > > > > > RC.
> > > > > > > > > > Any functional verification that you did on previous RCs
> should
> > > > > > > > therefore
> > > > > > > > > > easily carry over to this one.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
>

Tzu-Li (Gordon) Tai

Re: [VOTE] Release 1.8.0, release candidate #4

+1 (binding)

Functional checks:

- Built Flink from source (`mvn clean verify`) locally, with success
- Ran end-to-end tests locally for 5 times in a loop, no attempts failed
(Hadoop 2.8.4, Scala 2.12)
- Manually tested state schema evolution for POJO. Besides the tests that
@Congxian already did, additionally tested evolution cases with POJO
subclasses + non-registered POJOs.
- Manually tested migration of Scala stateful jobs that use case classes /
Scala collections as state types, performing the migration across Scala
2.11 to Scala 2.12.
- Reviewed release announcement PR

Misc / legal checks:

- checked checksums and signatures
- No binaries in source distribution
- Staging area does not seem to have any missing artifacts

Cheers,
Gordon

On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> @Shaoxuan
>
> The drop in the serializerAvro benchmark, as explained earlier in previous
> voting threads of earlier RCs, was due to a slower job initialization phase
> caused by slower deserialization of the AvroSerializer.
> Piotr also pointed out that after the number of records was increased in
> the serializer benchmarks, this drop was no longer observable before /
> after the changes in mid February.
> IMO, this is not critical as it does not affect the per-record performance
> / throughput, and therefore should not block this release.
>
> On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <[hidden email]>
> wrote:
>
>> By now, I'm reasonably sure that the test instabilities on the end-to-end
>> test are only instabilities. I pushed changes to increase timeouts to make
>> the tests more stable. As in any project, there will always be bugs but I
>> think we could release this RC4 and be reasonably sure that it works well.
>>
>> Now, we only need to have the required number of PMC votes.
>>
>> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
>> > +1 （non-binding)
>> >
>> > • checked signature and checksum ok
>> > • mvn clean package -DskipTests ok
>> > • Run job on yarn ok
>> > • Test state migration with POJO type (both heap and rocksdb) ok
>> > • - 1.6 -> 1.8
>> > • - 1.7 -> 1.8
>> > • - 1.8 -> 1.8
>> >
>> >
>> > Best, Congxian
>> > On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>, wrote:
>> > > +1 (non-binding)
>> > >
>> > > - checked JIRA release note
>> > > - ran "mvn package -DskipTests"
>> > > - checked signature and checksum
>> > > - started a cluster locally and ran some examples in binary
>> > > - checked web site announcement's PR
>> > >
>> > > Best,
>> > > Vino
>> > >
>> > >
>> > > Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
>> > >
>> > > > +1 (non-binding)
>> > > >
>> > > > - checked checksums and GPG files
>> > > > - build from source successfully- run end-to-end precommit tests
>> > > > successfully- run end-to-end nightly tests successfully
>> > > > Xiaowei
>> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
>> [hidden email]>
>> > > > wrote:
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > - Checked release notes: OK
>> > > > - Checked sums and signatures: OK
>> > > > - Source release
>> > > > - contains no binaries: OK
>> > > > - contains no 1.8-SNAPSHOT references: OK
>> > > > - build from source: OK (8u101)
>> > > > - mvn clean verify: OK (8u101)
>> > > > - Binary release
>> > > > - no examples appear to be missing
>> > > > - started a cluster; WebUI reachable, example ran successfully
>> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
>> > > > - Repository appears to contain all expected artifacts
>> > > >
>> > > > Best Regards,
>> > > > Yu
>> > > >
>> > > >
>> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]> wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > Checked items:
>> > > > > - checked checksums and GPG files
>> > > > > - verified that the source archives do not contains any binaries
>> > > > > - checked that all POM files point to the same version
>> > > > > - build from source successfully
>> > > > >
>> > > > > Best,
>> > > > > Kurt
>> > > > >
>> > > > >
>> > > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
>> [hidden email]>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > I tested RC4 with the following items:
>> > > > > > - Maven Central Repository contains all artifacts
>> > > > > > - Built the source with Maven (ensured all source files have
>> Apache
>> > > > > > headers), and executed built-in tests via "mvn clean verify"
>> > > > > > - Manually executed the tests in IntelliJ IDE
>> > > > > > - Verify that the quickstarts for Scala and Java are working
>> with the
>> > > > > > staging repository in IntelliJ
>> > > > > > - Checked the benchmark results. The perf regression of
>> > > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the
>> regression
>> > > > on
>> > > > > > serializer still exists.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Shaoxuan
>> > > > > >
>> > > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
>> [hidden email]
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Aljoscha, I think you are right, increase the timeout
>> config will
>> > > > > fix
>> > > > > > > this issue. this depends on the resource of Travis. I would
>> like
>> > > > share
>> > > > > > > some phenomenon during my test (not the flink problem) as
>> follows:
>> > > > :-)
>> > > > > > >
>> > > > > > > During my testing, `mvn clean verify` and `nightly end-to-end
>> test `
>> > > > > both
>> > > > > > > consume a lot of machine resources (especially
>> memory/network), and
>> > > > the
>> > > > > > > network bandwidth requirements of `nightly end-to-end test `
>> are also
>> > > > > > very
>> > > > > > > high. In China, need to use VPN acceleration (100~200Kb before
>> > > > > > > acceleration, 3~4Mb after acceleration), I have encountered:
>> [Avro
>> > > > > > > Confluent Schema Registry nightly end-to-end test' failed
>> after 18
>> > > > > > minutes
>> > > > > > > and 15 seconds! Test exited with exit Code 1] takes more than
>> 18
>> > > > > minutes,
>> > > > > > > the download failed because the network bandwidth is not
>> enough. and
>> > > > it
>> > > > > > > runs smoothly when using VPN acceleration. The overall
>> end-to-end run
>> > > > > was
>> > > > > > > passed twice. The Docker resource configuration (CUPs 7, Mem:
>> 28.7G,
>> > > > > > Swap:
>> > > > > > > 3.5G). See detail log here
>> > > > > > > <
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
>> > > > > > > >
>> > > > > > > .
>> > > > > > >
>> > > > > > > Just now, I had checked the Travis for your last commit
>> (Increase
>> > > > > startup
>> > > > > > > timeout in end-to-end tests), in addition to the Cleanup
>> phase, other
>> > > > > > > phases are successful. here
>> > > > > > > <https://travis-ci.org/apache/flink/builds/511071777>
>> > > > > > >
>> > > > > > > In order to verify that our speculation is accurate, I can
>> help with
>> > > > 10
>> > > > > > and
>> > > > > > > 20 seconds timeout config on my repo verification to see if
>> 100%
>> > > > > > recurring
>> > > > > > > timeout problem. It is already running, we are waiting for the
>> > > > result.
>> > > > > > > 10seconds <
>> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235749
>> > > > > >
>> > > > > > > 20seconds <
>> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235598
>> > > > > >
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Jincheng
>> > > > > > >
>> > > > > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
>> 上午1:04写道：
>> > > > > > >
>> > > > > > > > Thanks for the testing done so far!
>> > > > > > > >
>> > > > > > > > There has been quite some flakiness on Travis lately, see
>> here:
>> > > > > > > > https://travis-ci.org/apache/flink/branches <
>> > > > > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit
>> hesitant
>> > > > to
>> > > > > > > > release in this state. Looking at the tests you can see
>> that all of
>> > > > > the
>> > > > > > > > end-to-end tests fail because waiting for the dispatcher to
>> come up
>> > > > > > times
>> > > > > > > > out. I also noticed that this usually takes about 5-8
>> seconds on
>> > > > > > Travis,
>> > > > > > > so
>> > > > > > > > a 10 second timeout might be a bit low. I pushed commits to
>> > > > increase
>> > > > > > that
>> > > > > > > > to 20 secs. Let’s see what will happen.
>> > > > > > > >
>> > > > > > > > I’ll keep you posted!
>> > > > > > > > Aljoscha
>> > > > > > > >
>> > > > > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
>> > > > [hidden email]>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Great thanks for preparing the RC4 of Flink 1.8.0,
>> Aljoscha!
>> > > > > > > > >
>> > > > > > > > > +1 (non-binding)
>> > > > > > > > >
>> > > > > > > > > I checked the functional things as follows(Without
>> performance
>> > > > > > > > > verification):
>> > > > > > > > >
>> > > > > > > > > 1. Checking Artifacts：
>> > > > > > > > >
>> > > > > > > > > 1). Download the release source code - SUCCESS
>> > > > > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 -
>> SUCCESS
>> > > > > > > > > 3). Download the released JAR - SUCCESS
>> > > > > > > > > 4). Check if checksums and GPG files match the
>> corresponding
>> > > > > > release
>> > > > > > > > > files - SUCCESS.
>> > > > > > > > > 5). Verify that the source archives do not contain any
>> > > > binaries
>> > > > > -
>> > > > > > > > > SUCCESS.
>> > > > > > > > > 6). Build the source with `mvn clean verify -DskipTests`
>> to
>> > > > > ensure
>> > > > > > > all
>> > > > > > > > > source files have Apache headers - SUCCESS
>> > > > > > > > > 7). Check that all POM files point to the same version -
>> > > > SUCCESS
>> > > > > > > > > 8). Read the `README.md` file to ensure there is nothing
>> > > > > > unexpected
>> > > > > > > -
>> > > > > > > > > SUCCESS
>> > > > > > > > >
>> > > > > > > > > 2. Testing Larger Setups
>> > > > > > > > >
>> > > > > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
>> > > > > > > > > Testing Jobs: WordCount(Batch&Streaming),
>> > > > > > > DataStreamAllroundTestProgram
>> > > > > > > > >
>> > > > > > > > > 1). Use local&hdfs file systems for checkpoints - SUCCESS
>> > > > > > > > > 2). Use hdfs file systems for input/output -SUCCESS
>> > > > > > > > > 3). Run examples on YARN(with or without session) -
>> SUCCESS
>> > > > > > > > > 4). Test failover and recovery. - SUCCESS
>> > > > > > > > > 5). Test incremental&non-incremental checkpoint - SUCCESS
>> > > > > > > > > 6). Test connector - kafka -SUCCESS
>> > > > > > > > >
>> > > > > > > > > 3. Testing Functionality
>> > > > > > > > >
>> > > > > > > > > 1). Built-in tests(linux&mac os)
>> > > > > > > > > - `mvn cealn verify` (some test timeout error and test
>> case
>> > > > > bug
>> > > > > > > see
>> > > > > > > > > FLINK-12001 <
>> https://issues.apache.org/jira/browse/FLINK-12001>,
>> > > > > all
>> > > > > > > of
>> > > > > > > > > them are not the blocker)
>> > > > > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
>> > > > > > > -DskipTests）
>> > > > > > > > > - SUCCESS
>> > > > > > > > > - Run the scripted nightly end-to-end test - SUCCESS
>> > > > > > > > >
>> > > > > > > > > 2). Quickstarts
>> > > > > > > > > - Verify that the quickstarts for Scala with the staging
>> > > > > > > repository
>> > > > > > > > > in IntelliJ - SUCCESS
>> > > > > > > > > - Verify that the quickstarts for Java with the staging
>> > > > > > repository
>> > > > > > > > in
>> > > > > > > > > IntelliJ - SUCCESS
>> > > > > > > > >
>> > > > > > > > > 3). Simple Starter Experience and Use Cases
>> > > > > > > > >
>> > > > > > > > > - run all examples from IntelliJ IDE - SUCCESS
>> > > > > > > > > - Start a local cluster and verify that the processes -
>> > > > > SUCCESS
>> > > > > > > > > a. Examine the *.out files (should be empty) and the log
>> > > > > files
>> > > > > > > > > (should contain no exceptions)
>> > > > > > > > > b. Test for Linux, MacOS
>> > > > > > > > > c. Shutdown and verify there are no exceptions in the log
>> > > > > > output
>> > > > > > > > > (after shutdown)
>> > > > > > > > >
>> > > > > > > > > - Verify that the examples are running from both
>> ./bin/flink
>> > > > > and
>> > > > > > > > from
>> > > > > > > > > the web-based job submission tool(following items) -
>> SUCCESS
>> > > > > > > > > a. Start multiple task managers in the local cluster
>> > > > > > > > > b. Change the flink-conf.yml to define more than one task
>> > > > > slot
>> > > > > > > (2)
>> > > > > > > > > c. Run the examples with a parallelism > 1
>> > > > > > > > > d. Examine the log output - no error messages should be
>> > > > > > > > encountered
>> > > > > > > > >
>> > > > > > > > > 4. Review the PR
>> > > > > > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated
>> the
>> > > > > > > release
>> > > > > > > > > date to correct date before merging.
>> > > > > > > > >
>> > > > > > > > > Cheers,
>> > > > > > > > > Jincheng
>> > > > > > > > >
>> > > > > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一
>> 下午4:11写道：
>> > > > > > > > >
>> > > > > > > > > > +1 from my side. Previously spotted performance
>> regression seems
>> > > > > to
>> > > > > > be
>> > > > > > > > > > gone, or mostly gone.
>> > > > > > > > > >
>> > > > > > > > > > Piotrek
>> > > > > > > > > >
>> > > > > > > > > > > On 21 Mar 2019, at 17:52, Aljoscha Krettek <
>> > > > [hidden email]>
>> > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Hi everyone,
>> > > > > > > > > > > Please review and vote on the release candidate 4 for
>> Flink
>> > > > > 1.8.0,
>> > > > > > as
>> > > > > > > > > > follows:
>> > > > > > > > > > > [ ] +1, Approve the release
>> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
>> specific
>> > > > > > comments)
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > The complete staging area is available for your
>> review, which
>> > > > > > > includes:
>> > > > > > > > > > > * JIRA release notes [1],
>> > > > > > > > > > > * the official Apache source release and binary
>> convenience
>> > > > > > releases
>> > > > > > > to
>> > > > > > > > > > be deployed to dist.apache.org [2], which are signed
>> with the
>> > > > key
>> > > > > > > with
>> > > > > > > > > > fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
>> [3],
>> > > > > > > > > > > * all artifacts to be deployed to the Maven Central
>> Repository
>> > > > > [4],
>> > > > > > > > > > > * source code tag "release-1.8.0-rc4" [5],
>> > > > > > > > > > > * website pull request listing the new release [6]
>> > > > > > > > > > > * website pull request adding announcement blog post
>> [7].
>> > > > > > > > > > >
>> > > > > > > > > > > The vote will be open for at least 72 hours. It is
>> adopted by
>> > > > > > > majority
>> > > > > > > > > > approval, with at least 3 PMC affirmative votes.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > > Aljoscha
>> > > > > > > > > > >
>> > > > > > > > > > > [1]
>> > > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
>> > > > > > > > > > > [2]
>> > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
>> > > > > > > > > > > [3]
>> https://dist.apache.org/repos/dist/release/flink/KEYS
>> > > > > > > > > > > [4]
>> > > > > > > > > >
>> > > > > > >
>> > > >
>> https://repository.apache.org/content/repositories/orgapacheflink-1215
>> > > > > > > > > > > [5]
>> > > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
>> > > > > > > > > > > [6] https://github.com/apache/flink-web/pull/180 <
>> > > > > > > > > > https://github.com/apache/flink-web/pull/180>
>> > > > > > > > > > > [7] https://github.com/apache/flink-web/pull/179 <
>> > > > > > > > > > https://github.com/apache/flink-web/pull/179>
>> > > > > > > > > > >
>> > > > > > > > > > > P.S. The difference to the previous RCs is small, you
>> can fetch
>> > > > > the
>> > > > > > > > tags
>> > > > > > > > > > and do a "git log release-1.8.0-rc1..release-1.8.0-rc4”
>> to see
>> > > > the
>> > > > > > > > > > difference in commits. Its fixes for the issues that
>> led to the
>> > > > > > > > > > cancellation of the previous RCs plus smaller fixes.
>> Most
>> > > > > > > > > > verification/testing that was carried out should apply
>> as is to
>> > > > > this
>> > > > > > > RC.
>> > > > > > > > > > Any functional verification that you did on previous
>> RCs should
>> > > > > > > > therefore
>> > > > > > > > > > easily carry over to this one.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> >
>>
>

Richard Deurwaarder

Re: [VOTE] Release 1.8.0, release candidate #4

-1 (non-binding)

- Ran integration tests locally (1000+) of our flink job, all succeeded.
- Attempted to run job on hadoop, failed. It failed because we have a
firewall in place and we cannot set the rest port to a specific port/port
range.
Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
setting a REST port when running on yarn (
https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
)
Code-wise it seems rather straightforward to fix but I am unsure about the
reason why this is hard-coded to 0 and what the impact would be.

It would benefit us greatly if a fix for this could make it to 1.8.0.

Regards,

Richard

On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> +1 (binding)
>
> Functional checks:
>
> - Built Flink from source (`mvn clean verify`) locally, with success
> - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
> (Hadoop 2.8.4, Scala 2.12)
> - Manually tested state schema evolution for POJO. Besides the tests that
> @Congxian already did, additionally tested evolution cases with POJO
> subclasses + non-registered POJOs.
> - Manually tested migration of Scala stateful jobs that use case classes /
> Scala collections as state types, performing the migration across Scala
> 2.11 to Scala 2.12.
> - Reviewed release announcement PR
>
> Misc / legal checks:
>
> - checked checksums and signatures
> - No binaries in source distribution
> - Staging area does not seem to have any missing artifacts
>
> Cheers,
> Gordon
>
> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> > @Shaoxuan
> >
> > The drop in the serializerAvro benchmark, as explained earlier in
> previous
> > voting threads of earlier RCs, was due to a slower job initialization
> phase
> > caused by slower deserialization of the AvroSerializer.
> > Piotr also pointed out that after the number of records was increased in
> > the serializer benchmarks, this drop was no longer observable before /
> > after the changes in mid February.
> > IMO, this is not critical as it does not affect the per-record
> performance
> > / throughput, and therefore should not block this release.
> >
> > On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <[hidden email]>
> > wrote:
> >
> >> By now, I'm reasonably sure that the test instabilities on the
> end-to-end
> >> test are only instabilities. I pushed changes to increase timeouts to
> make
> >> the tests more stable. As in any project, there will always be bugs but
> I
> >> think we could release this RC4 and be reasonably sure that it works
> well.
> >>
> >> Now, we only need to have the required number of PMC votes.
> >>
> >> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> >> > +1 （non-binding)
> >> >
> >> > • checked signature and checksum ok
> >> > • mvn clean package -DskipTests ok
> >> > • Run job on yarn ok
> >> > • Test state migration with POJO type (both heap and rocksdb) ok
> >> > • - 1.6 -> 1.8
> >> > • - 1.7 -> 1.8
> >> > • - 1.8 -> 1.8
> >> >
> >> >
> >> > Best, Congxian
> >> > On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>,
> wrote:
> >> > > +1 (non-binding)
> >> > >
> >> > > - checked JIRA release note
> >> > > - ran "mvn package -DskipTests"
> >> > > - checked signature and checksum
> >> > > - started a cluster locally and ran some examples in binary
> >> > > - checked web site announcement's PR
> >> > >
> >> > > Best,
> >> > > Vino
> >> > >
> >> > >
> >> > > Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
> >> > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > - checked checksums and GPG files
> >> > > > - build from source successfully- run end-to-end precommit tests
> >> > > > successfully- run end-to-end nightly tests successfully
> >> > > > Xiaowei
> >> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> >> [hidden email]>
> >> > > > wrote:
> >> > > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > - Checked release notes: OK
> >> > > > - Checked sums and signatures: OK
> >> > > > - Source release
> >> > > > - contains no binaries: OK
> >> > > > - contains no 1.8-SNAPSHOT references: OK
> >> > > > - build from source: OK (8u101)
> >> > > > - mvn clean verify: OK (8u101)
> >> > > > - Binary release
> >> > > > - no examples appear to be missing
> >> > > > - started a cluster; WebUI reachable, example ran successfully
> >> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> >> > > > - Repository appears to contain all expected artifacts
> >> > > >
> >> > > > Best Regards,
> >> > > > Yu
> >> > > >
> >> > > >
> >> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]>
> wrote:
> >> > > >
> >> > > > > +1 (non-binding)
> >> > > > >
> >> > > > > Checked items:
> >> > > > > - checked checksums and GPG files
> >> > > > > - verified that the source archives do not contains any binaries
> >> > > > > - checked that all POM files point to the same version
> >> > > > > - build from source successfully
> >> > > > >
> >> > > > > Best,
> >> > > > > Kurt
> >> > > > >
> >> > > > >
> >> > > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
> >> [hidden email]>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1 (non-binding)
> >> > > > > >
> >> > > > > > I tested RC4 with the following items:
> >> > > > > > - Maven Central Repository contains all artifacts
> >> > > > > > - Built the source with Maven (ensured all source files have
> >> Apache
> >> > > > > > headers), and executed built-in tests via "mvn clean verify"
> >> > > > > > - Manually executed the tests in IntelliJ IDE
> >> > > > > > - Verify that the quickstarts for Scala and Java are working
> >> with the
> >> > > > > > staging repository in IntelliJ
> >> > > > > > - Checked the benchmark results. The perf regression of
> >> > > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the
> >> regression
> >> > > > on
> >> > > > > > serializer still exists.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > Shaoxuan
> >> > > > > >
> >> > > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
> >> [hidden email]
> >> > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Aljoscha, I think you are right, increase the timeout
> >> config will
> >> > > > > fix
> >> > > > > > > this issue. this depends on the resource of Travis. I would
> >> like
> >> > > > share
> >> > > > > > > some phenomenon during my test (not the flink problem) as
> >> follows:
> >> > > > :-)
> >> > > > > > >
> >> > > > > > > During my testing, `mvn clean verify` and `nightly
> end-to-end
> >> test `
> >> > > > > both
> >> > > > > > > consume a lot of machine resources (especially
> >> memory/network), and
> >> > > > the
> >> > > > > > > network bandwidth requirements of `nightly end-to-end test `
> >> are also
> >> > > > > > very
> >> > > > > > > high. In China, need to use VPN acceleration (100~200Kb
> before
> >> > > > > > > acceleration, 3~4Mb after acceleration), I have encountered:
> >> [Avro
> >> > > > > > > Confluent Schema Registry nightly end-to-end test' failed
> >> after 18
> >> > > > > > minutes
> >> > > > > > > and 15 seconds! Test exited with exit Code 1] takes more
> than
> >> 18
> >> > > > > minutes,
> >> > > > > > > the download failed because the network bandwidth is not
> >> enough. and
> >> > > > it
> >> > > > > > > runs smoothly when using VPN acceleration. The overall
> >> end-to-end run
> >> > > > > was
> >> > > > > > > passed twice. The Docker resource configuration (CUPs 7,
> Mem:
> >> 28.7G,
> >> > > > > > Swap:
> >> > > > > > > 3.5G). See detail log here
> >> > > > > > > <
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> >> > > > > > > >
> >> > > > > > > .
> >> > > > > > >
> >> > > > > > > Just now, I had checked the Travis for your last commit
> >> (Increase
> >> > > > > startup
> >> > > > > > > timeout in end-to-end tests), in addition to the Cleanup
> >> phase, other
> >> > > > > > > phases are successful. here
> >> > > > > > > <https://travis-ci.org/apache/flink/builds/511071777>
> >> > > > > > >
> >> > > > > > > In order to verify that our speculation is accurate, I can
> >> help with
> >> > > > 10
> >> > > > > > and
> >> > > > > > > 20 seconds timeout config on my repo verification to see if
> >> 100%
> >> > > > > > recurring
> >> > > > > > > timeout problem. It is already running, we are waiting for
> the
> >> > > > result.
> >> > > > > > > 10seconds <
> >> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235749
> >> > > > > >
> >> > > > > > > 20seconds <
> >> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235598
> >> > > > > >
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Jincheng
> >> > > > > > >
> >> > > > > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
> >> 上午1:04写道：
> >> > > > > > >
> >> > > > > > > > Thanks for the testing done so far!
> >> > > > > > > >
> >> > > > > > > > There has been quite some flakiness on Travis lately, see
> >> here:
> >> > > > > > > > https://travis-ci.org/apache/flink/branches <
> >> > > > > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit
> >> hesitant
> >> > > > to
> >> > > > > > > > release in this state. Looking at the tests you can see
> >> that all of
> >> > > > > the
> >> > > > > > > > end-to-end tests fail because waiting for the dispatcher
> to
> >> come up
> >> > > > > > times
> >> > > > > > > > out. I also noticed that this usually takes about 5-8
> >> seconds on
> >> > > > > > Travis,
> >> > > > > > > so
> >> > > > > > > > a 10 second timeout might be a bit low. I pushed commits
> to
> >> > > > increase
> >> > > > > > that
> >> > > > > > > > to 20 secs. Let’s see what will happen.
> >> > > > > > > >
> >> > > > > > > > I’ll keep you posted!
> >> > > > > > > > Aljoscha
> >> > > > > > > >
> >> > > > > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
> >> > > > [hidden email]>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > Great thanks for preparing the RC4 of Flink 1.8.0,
> >> Aljoscha!
> >> > > > > > > > >
> >> > > > > > > > > +1 (non-binding)
> >> > > > > > > > >
> >> > > > > > > > > I checked the functional things as follows(Without
> >> performance
> >> > > > > > > > > verification):
> >> > > > > > > > >
> >> > > > > > > > > 1. Checking Artifacts：
> >> > > > > > > > >
> >> > > > > > > > > 1). Download the release source code - SUCCESS
> >> > > > > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 -
> >> SUCCESS
> >> > > > > > > > > 3). Download the released JAR - SUCCESS
> >> > > > > > > > > 4). Check if checksums and GPG files match the
> >> corresponding
> >> > > > > > release
> >> > > > > > > > > files - SUCCESS.
> >> > > > > > > > > 5). Verify that the source archives do not contain any
> >> > > > binaries
> >> > > > > -
> >> > > > > > > > > SUCCESS.
> >> > > > > > > > > 6). Build the source with `mvn clean verify -DskipTests`
> >> to
> >> > > > > ensure
> >> > > > > > > all
> >> > > > > > > > > source files have Apache headers - SUCCESS
> >> > > > > > > > > 7). Check that all POM files point to the same version -
> >> > > > SUCCESS
> >> > > > > > > > > 8). Read the `README.md` file to ensure there is nothing
> >> > > > > > unexpected
> >> > > > > > > -
> >> > > > > > > > > SUCCESS
> >> > > > > > > > >
> >> > > > > > > > > 2. Testing Larger Setups
> >> > > > > > > > >
> >> > > > > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> >> > > > > > > > > Testing Jobs: WordCount(Batch&Streaming),
> >> > > > > > > DataStreamAllroundTestProgram
> >> > > > > > > > >
> >> > > > > > > > > 1). Use local&hdfs file systems for checkpoints -
> SUCCESS
> >> > > > > > > > > 2). Use hdfs file systems for input/output -SUCCESS
> >> > > > > > > > > 3). Run examples on YARN(with or without session) -
> >> SUCCESS
> >> > > > > > > > > 4). Test failover and recovery. - SUCCESS
> >> > > > > > > > > 5). Test incremental&non-incremental checkpoint -
> SUCCESS
> >> > > > > > > > > 6). Test connector - kafka -SUCCESS
> >> > > > > > > > >
> >> > > > > > > > > 3. Testing Functionality
> >> > > > > > > > >
> >> > > > > > > > > 1). Built-in tests(linux&mac os)
> >> > > > > > > > > - `mvn cealn verify` (some test timeout error and test
> >> case
> >> > > > > bug
> >> > > > > > > see
> >> > > > > > > > > FLINK-12001 <
> >> https://issues.apache.org/jira/browse/FLINK-12001>,
> >> > > > > all
> >> > > > > > > of
> >> > > > > > > > > them are not the blocker)
> >> > > > > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> >> > > > > > > -DskipTests）
> >> > > > > > > > > - SUCCESS
> >> > > > > > > > > - Run the scripted nightly end-to-end test - SUCCESS
> >> > > > > > > > >
> >> > > > > > > > > 2). Quickstarts
> >> > > > > > > > > - Verify that the quickstarts for Scala with the staging
> >> > > > > > > repository
> >> > > > > > > > > in IntelliJ - SUCCESS
> >> > > > > > > > > - Verify that the quickstarts for Java with the staging
> >> > > > > > repository
> >> > > > > > > > in
> >> > > > > > > > > IntelliJ - SUCCESS
> >> > > > > > > > >
> >> > > > > > > > > 3). Simple Starter Experience and Use Cases
> >> > > > > > > > >
> >> > > > > > > > > - run all examples from IntelliJ IDE - SUCCESS
> >> > > > > > > > > - Start a local cluster and verify that the processes -
> >> > > > > SUCCESS
> >> > > > > > > > > a. Examine the *.out files (should be empty) and the log
> >> > > > > files
> >> > > > > > > > > (should contain no exceptions)
> >> > > > > > > > > b. Test for Linux, MacOS
> >> > > > > > > > > c. Shutdown and verify there are no exceptions in the
> log
> >> > > > > > output
> >> > > > > > > > > (after shutdown)
> >> > > > > > > > >
> >> > > > > > > > > - Verify that the examples are running from both
> >> ./bin/flink
> >> > > > > and
> >> > > > > > > > from
> >> > > > > > > > > the web-based job submission tool(following items) -
> >> SUCCESS
> >> > > > > > > > > a. Start multiple task managers in the local cluster
> >> > > > > > > > > b. Change the flink-conf.yml to define more than one
> task
> >> > > > > slot
> >> > > > > > > (2)
> >> > > > > > > > > c. Run the examples with a parallelism > 1
> >> > > > > > > > > d. Examine the log output - no error messages should be
> >> > > > > > > > encountered
> >> > > > > > > > >
> >> > > > > > > > > 4. Review the PR
> >> > > > > > > > > - [Add 1.8 Release Blog Post] - Just a reminder, updated
> >> the
> >> > > > > > > release
> >> > > > > > > > > date to correct date before merging.
> >> > > > > > > > >
> >> > > > > > > > > Cheers,
> >> > > > > > > > > Jincheng
> >> > > > > > > > >
> >> > > > > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一
> >> 下午4:11写道：
> >> > > > > > > > >
> >> > > > > > > > > > +1 from my side. Previously spotted performance
> >> regression seems
> >> > > > > to
> >> > > > > > be
> >> > > > > > > > > > gone, or mostly gone.
> >> > > > > > > > > >
> >> > > > > > > > > > Piotrek
> >> > > > > > > > > >
> >> > > > > > > > > > > On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> >> > > > [hidden email]>
> >> > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > > Please review and vote on the release candidate 4
> for
> >> Flink
> >> > > > > 1.8.0,
> >> > > > > > as
> >> > > > > > > > > > follows:
> >> > > > > > > > > > > [ ] +1, Approve the release
> >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> >> specific
> >> > > > > > comments)
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > The complete staging area is available for your
> >> review, which
> >> > > > > > > includes:
> >> > > > > > > > > > > * JIRA release notes [1],
> >> > > > > > > > > > > * the official Apache source release and binary
> >> convenience
> >> > > > > > releases
> >> > > > > > > to
> >> > > > > > > > > > be deployed to dist.apache.org [2], which are signed
> >> with the
> >> > > > key
> >> > > > > > > with
> >> > > > > > > > > > fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
> >> [3],
> >> > > > > > > > > > > * all artifacts to be deployed to the Maven Central
> >> Repository
> >> > > > > [4],
> >> > > > > > > > > > > * source code tag "release-1.8.0-rc4" [5],
> >> > > > > > > > > > > * website pull request listing the new release [6]
> >> > > > > > > > > > > * website pull request adding announcement blog post
> >> [7].
> >> > > > > > > > > > >
> >> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> >> adopted by
> >> > > > > > > majority
> >> > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > > Aljoscha
> >> > > > > > > > > > >
> >> > > > > > > > > > > [1]
> >> > > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> >> > > > > > > > > > > [2]
> >> > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> >> > > > > > > > > > > [3]
> >> https://dist.apache.org/repos/dist/release/flink/KEYS
> >> > > > > > > > > > > [4]
> >> > > > > > > > > >
> >> > > > > > >
> >> > > >
> >> https://repository.apache.org/content/repositories/orgapacheflink-1215
> >> > > > > > > > > > > [5]
> >> > > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> >> > > > > > > > > > > [6] https://github.com/apache/flink-web/pull/180 <
> >> > > > > > > > > > https://github.com/apache/flink-web/pull/180>
> >> > > > > > > > > > > [7] https://github.com/apache/flink-web/pull/179 <
> >> > > > > > > > > > https://github.com/apache/flink-web/pull/179>
> >> > > > > > > > > > >
> >> > > > > > > > > > > P.S. The difference to the previous RCs is small,
> you
> >> can fetch
> >> > > > > the
> >> > > > > > > > tags
> >> > > > > > > > > > and do a "git log
> release-1.8.0-rc1..release-1.8.0-rc4”
> >> to see
> >> > > > the
> >> > > > > > > > > > difference in commits. Its fixes for the issues that
> >> led to the
> >> > > > > > > > > > cancellation of the previous RCs plus smaller fixes.
> >> Most
> >> > > > > > > > > > verification/testing that was carried out should apply
> >> as is to
> >> > > > > this
> >> > > > > > > RC.
> >> > > > > > > > > > Any functional verification that you did on previous
> >> RCs should
> >> > > > > > > > therefore
> >> > > > > > > > > > easily carry over to this one.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> >
> >>
> >
>

Tzu-Li (Gordon) Tai

Re: [VOTE] Release 1.8.0, release candidate #4

@Yu discovered this issue, which IMO is probably a blocker for the
release: https://issues.apache.org/jira/browse/FLINK-12064.
The bug is a regression caused by previous state backend refactorings,
which can result in incorrect representation of the schema of serialized
keys in state because the wrong key serializer instance is being
snapshotted.

There is already a PR to fix this, I'll try to review and merge it over the
weekend.

On Fri, Mar 29, 2019 at 7:13 AM Richard Deurwaarder <[hidden email]> wrote:

> -1 (non-binding)
>
> - Ran integration tests locally (1000+) of our flink job, all succeeded.
> - Attempted to run job on hadoop, failed. It failed because we have a
> firewall in place and we cannot set the rest port to a specific port/port
> range.
> Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
> setting a REST port when running on yarn (
>
> https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
> )
> Code-wise it seems rather straightforward to fix but I am unsure about the
> reason why this is hard-coded to 0 and what the impact would be.
>
> It would benefit us greatly if a fix for this could make it to 1.8.0.
>
> Regards,
>
> Richard
>
> On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
> > +1 (binding)
> >
> > Functional checks:
> >
> > - Built Flink from source (`mvn clean verify`) locally, with success
> > - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
> > (Hadoop 2.8.4, Scala 2.12)
> > - Manually tested state schema evolution for POJO. Besides the tests that
> > @Congxian already did, additionally tested evolution cases with POJO
> > subclasses + non-registered POJOs.
> > - Manually tested migration of Scala stateful jobs that use case classes
> /
> > Scala collections as state types, performing the migration across Scala
> > 2.11 to Scala 2.12.
> > - Reviewed release announcement PR
> >
> > Misc / legal checks:
> >
> > - checked checksums and signatures
> > - No binaries in source distribution
> > - Staging area does not seem to have any missing artifacts
> >
> > Cheers,
> > Gordon
> >
> > On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <[hidden email]
> >
> > wrote:
> >
> > > @Shaoxuan
> > >
> > > The drop in the serializerAvro benchmark, as explained earlier in
> > previous
> > > voting threads of earlier RCs, was due to a slower job initialization
> > phase
> > > caused by slower deserialization of the AvroSerializer.
> > > Piotr also pointed out that after the number of records was increased
> in
> > > the serializer benchmarks, this drop was no longer observable before /
> > > after the changes in mid February.
> > > IMO, this is not critical as it does not affect the per-record
> > performance
> > > / throughput, and therefore should not block this release.
> > >
> > > On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <
> [hidden email]>
> > > wrote:
> > >
> > >> By now, I'm reasonably sure that the test instabilities on the
> > end-to-end
> > >> test are only instabilities. I pushed changes to increase timeouts to
> > make
> > >> the tests more stable. As in any project, there will always be bugs
> but
> > I
> > >> think we could release this RC4 and be reasonably sure that it works
> > well.
> > >>
> > >> Now, we only need to have the required number of PMC votes.
> > >>
> > >> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> > >> > +1 （non-binding)
> > >> >
> > >> > • checked signature and checksum ok
> > >> > • mvn clean package -DskipTests ok
> > >> > • Run job on yarn ok
> > >> > • Test state migration with POJO type (both heap and rocksdb) ok
> > >> > • - 1.6 -> 1.8
> > >> > • - 1.7 -> 1.8
> > >> > • - 1.8 -> 1.8
> > >> >
> > >> >
> > >> > Best, Congxian
> > >> > On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>,
> > wrote:
> > >> > > +1 (non-binding)
> > >> > >
> > >> > > - checked JIRA release note
> > >> > > - ran "mvn package -DskipTests"
> > >> > > - checked signature and checksum
> > >> > > - started a cluster locally and ran some examples in binary
> > >> > > - checked web site announcement's PR
> > >> > >
> > >> > > Best,
> > >> > > Vino
> > >> > >
> > >> > >
> > >> > > Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
> > >> > >
> > >> > > > +1 (non-binding)
> > >> > > >
> > >> > > > - checked checksums and GPG files
> > >> > > > - build from source successfully- run end-to-end precommit tests
> > >> > > > successfully- run end-to-end nightly tests successfully
> > >> > > > Xiaowei
> > >> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> > >> [hidden email]>
> > >> > > > wrote:
> > >> > > >
> > >> > > > +1 (non-binding)
> > >> > > >
> > >> > > > - Checked release notes: OK
> > >> > > > - Checked sums and signatures: OK
> > >> > > > - Source release
> > >> > > > - contains no binaries: OK
> > >> > > > - contains no 1.8-SNAPSHOT references: OK
> > >> > > > - build from source: OK (8u101)
> > >> > > > - mvn clean verify: OK (8u101)
> > >> > > > - Binary release
> > >> > > > - no examples appear to be missing
> > >> > > > - started a cluster; WebUI reachable, example ran successfully
> > >> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> > >> > > > - Repository appears to contain all expected artifacts
> > >> > > >
> > >> > > > Best Regards,
> > >> > > > Yu
> > >> > > >
> > >> > > >
> > >> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]>
> > wrote:
> > >> > > >
> > >> > > > > +1 (non-binding)
> > >> > > > >
> > >> > > > > Checked items:
> > >> > > > > - checked checksums and GPG files
> > >> > > > > - verified that the source archives do not contains any
> binaries
> > >> > > > > - checked that all POM files point to the same version
> > >> > > > > - build from source successfully
> > >> > > > >
> > >> > > > > Best,
> > >> > > > > Kurt
> > >> > > > >
> > >> > > > >
> > >> > > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
> > >> [hidden email]>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > +1 (non-binding)
> > >> > > > > >
> > >> > > > > > I tested RC4 with the following items:
> > >> > > > > > - Maven Central Repository contains all artifacts
> > >> > > > > > - Built the source with Maven (ensured all source files have
> > >> Apache
> > >> > > > > > headers), and executed built-in tests via "mvn clean verify"
> > >> > > > > > - Manually executed the tests in IntelliJ IDE
> > >> > > > > > - Verify that the quickstarts for Scala and Java are working
> > >> with the
> > >> > > > > > staging repository in IntelliJ
> > >> > > > > > - Checked the benchmark results. The perf regression of
> > >> > > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the
> > >> regression
> > >> > > > on
> > >> > > > > > serializer still exists.
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > > Shaoxuan
> > >> > > > > >
> > >> > > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
> > >> [hidden email]
> > >> > > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Aljoscha, I think you are right, increase the timeout
> > >> config will
> > >> > > > > fix
> > >> > > > > > > this issue. this depends on the resource of Travis. I
> would
> > >> like
> > >> > > > share
> > >> > > > > > > some phenomenon during my test (not the flink problem) as
> > >> follows:
> > >> > > > :-)
> > >> > > > > > >
> > >> > > > > > > During my testing, `mvn clean verify` and `nightly
> > end-to-end
> > >> test `
> > >> > > > > both
> > >> > > > > > > consume a lot of machine resources (especially
> > >> memory/network), and
> > >> > > > the
> > >> > > > > > > network bandwidth requirements of `nightly end-to-end
> test `
> > >> are also
> > >> > > > > > very
> > >> > > > > > > high. In China, need to use VPN acceleration (100~200Kb
> > before
> > >> > > > > > > acceleration, 3~4Mb after acceleration), I have
> encountered:
> > >> [Avro
> > >> > > > > > > Confluent Schema Registry nightly end-to-end test' failed
> > >> after 18
> > >> > > > > > minutes
> > >> > > > > > > and 15 seconds! Test exited with exit Code 1] takes more
> > than
> > >> 18
> > >> > > > > minutes,
> > >> > > > > > > the download failed because the network bandwidth is not
> > >> enough. and
> > >> > > > it
> > >> > > > > > > runs smoothly when using VPN acceleration. The overall
> > >> end-to-end run
> > >> > > > > was
> > >> > > > > > > passed twice. The Docker resource configuration (CUPs 7,
> > Mem:
> > >> 28.7G,
> > >> > > > > > Swap:
> > >> > > > > > > 3.5G). See detail log here
> > >> > > > > > > <
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >>
> >
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > >> > > > > > > >
> > >> > > > > > > .
> > >> > > > > > >
> > >> > > > > > > Just now, I had checked the Travis for your last commit
> > >> (Increase
> > >> > > > > startup
> > >> > > > > > > timeout in end-to-end tests), in addition to the Cleanup
> > >> phase, other
> > >> > > > > > > phases are successful. here
> > >> > > > > > > <https://travis-ci.org/apache/flink/builds/511071777>
> > >> > > > > > >
> > >> > > > > > > In order to verify that our speculation is accurate, I can
> > >> help with
> > >> > > > 10
> > >> > > > > > and
> > >> > > > > > > 20 seconds timeout config on my repo verification to see
> if
> > >> 100%
> > >> > > > > > recurring
> > >> > > > > > > timeout problem. It is already running, we are waiting for
> > the
> > >> > > > result.
> > >> > > > > > > 10seconds <
> > >> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235749
> > >> > > > > >
> > >> > > > > > > 20seconds <
> > >> > > > https://travis-ci.org/sunjincheng121/flink/builds/511235598
> > >> > > > > >
> > >> > > > > > >
> > >> > > > > > > Best,
> > >> > > > > > > Jincheng
> > >> > > > > > >
> > >> > > > > > > Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
> > >> 上午1:04写道：
> > >> > > > > > >
> > >> > > > > > > > Thanks for the testing done so far!
> > >> > > > > > > >
> > >> > > > > > > > There has been quite some flakiness on Travis lately,
> see
> > >> here:
> > >> > > > > > > > https://travis-ci.org/apache/flink/branches <
> > >> > > > > > > > https://travis-ci.org/apache/flink/branches>. I’m a bit
> > >> hesitant
> > >> > > > to
> > >> > > > > > > > release in this state. Looking at the tests you can see
> > >> that all of
> > >> > > > > the
> > >> > > > > > > > end-to-end tests fail because waiting for the dispatcher
> > to
> > >> come up
> > >> > > > > > times
> > >> > > > > > > > out. I also noticed that this usually takes about 5-8
> > >> seconds on
> > >> > > > > > Travis,
> > >> > > > > > > so
> > >> > > > > > > > a 10 second timeout might be a bit low. I pushed commits
> > to
> > >> > > > increase
> > >> > > > > > that
> > >> > > > > > > > to 20 secs. Let’s see what will happen.
> > >> > > > > > > >
> > >> > > > > > > > I’ll keep you posted!
> > >> > > > > > > > Aljoscha
> > >> > > > > > > >
> > >> > > > > > > > > On 25. Mar 2019, at 13:13, jincheng sun <
> > >> > > > [hidden email]>
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > Great thanks for preparing the RC4 of Flink 1.8.0,
> > >> Aljoscha!
> > >> > > > > > > > >
> > >> > > > > > > > > +1 (non-binding)
> > >> > > > > > > > >
> > >> > > > > > > > > I checked the functional things as follows(Without
> > >> performance
> > >> > > > > > > > > verification):
> > >> > > > > > > > >
> > >> > > > > > > > > 1. Checking Artifacts：
> > >> > > > > > > > >
> > >> > > > > > > > > 1). Download the release source code - SUCCESS
> > >> > > > > > > > > 2). Check Source release flink-1.8.0-src.tgz.sha512 -
> > >> SUCCESS
> > >> > > > > > > > > 3). Download the released JAR - SUCCESS
> > >> > > > > > > > > 4). Check if checksums and GPG files match the
> > >> corresponding
> > >> > > > > > release
> > >> > > > > > > > > files - SUCCESS.
> > >> > > > > > > > > 5). Verify that the source archives do not contain any
> > >> > > > binaries
> > >> > > > > -
> > >> > > > > > > > > SUCCESS.
> > >> > > > > > > > > 6). Build the source with `mvn clean verify
> -DskipTests`
> > >> to
> > >> > > > > ensure
> > >> > > > > > > all
> > >> > > > > > > > > source files have Apache headers - SUCCESS
> > >> > > > > > > > > 7). Check that all POM files point to the same
> version -
> > >> > > > SUCCESS
> > >> > > > > > > > > 8). Read the `README.md` file to ensure there is
> nothing
> > >> > > > > > unexpected
> > >> > > > > > > -
> > >> > > > > > > > > SUCCESS
> > >> > > > > > > > >
> > >> > > > > > > > > 2. Testing Larger Setups
> > >> > > > > > > > >
> > >> > > > > > > > > Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > >> > > > > > > > > Testing Jobs: WordCount(Batch&Streaming),
> > >> > > > > > > DataStreamAllroundTestProgram
> > >> > > > > > > > >
> > >> > > > > > > > > 1). Use local&hdfs file systems for checkpoints -
> > SUCCESS
> > >> > > > > > > > > 2). Use hdfs file systems for input/output -SUCCESS
> > >> > > > > > > > > 3). Run examples on YARN(with or without session) -
> > >> SUCCESS
> > >> > > > > > > > > 4). Test failover and recovery. - SUCCESS
> > >> > > > > > > > > 5). Test incremental&non-incremental checkpoint -
> > SUCCESS
> > >> > > > > > > > > 6). Test connector - kafka -SUCCESS
> > >> > > > > > > > >
> > >> > > > > > > > > 3. Testing Functionality
> > >> > > > > > > > >
> > >> > > > > > > > > 1). Built-in tests(linux&mac os)
> > >> > > > > > > > > - `mvn cealn verify` (some test timeout error and test
> > >> case
> > >> > > > > bug
> > >> > > > > > > see
> > >> > > > > > > > > FLINK-12001 <
> > >> https://issues.apache.org/jira/browse/FLINK-12001>,
> > >> > > > > all
> > >> > > > > > > of
> > >> > > > > > > > > them are not the blocker)
> > >> > > > > > > > > - build for scala 2.11（mvn clean install -P scala-2.11
> > >> > > > > > > -DskipTests）
> > >> > > > > > > > > - SUCCESS
> > >> > > > > > > > > - Run the scripted nightly end-to-end test - SUCCESS
> > >> > > > > > > > >
> > >> > > > > > > > > 2). Quickstarts
> > >> > > > > > > > > - Verify that the quickstarts for Scala with the
> staging
> > >> > > > > > > repository
> > >> > > > > > > > > in IntelliJ - SUCCESS
> > >> > > > > > > > > - Verify that the quickstarts for Java with the
> staging
> > >> > > > > > repository
> > >> > > > > > > > in
> > >> > > > > > > > > IntelliJ - SUCCESS
> > >> > > > > > > > >
> > >> > > > > > > > > 3). Simple Starter Experience and Use Cases
> > >> > > > > > > > >
> > >> > > > > > > > > - run all examples from IntelliJ IDE - SUCCESS
> > >> > > > > > > > > - Start a local cluster and verify that the processes
> -
> > >> > > > > SUCCESS
> > >> > > > > > > > > a. Examine the *.out files (should be empty) and the
> log
> > >> > > > > files
> > >> > > > > > > > > (should contain no exceptions)
> > >> > > > > > > > > b. Test for Linux, MacOS
> > >> > > > > > > > > c. Shutdown and verify there are no exceptions in the
> > log
> > >> > > > > > output
> > >> > > > > > > > > (after shutdown)
> > >> > > > > > > > >
> > >> > > > > > > > > - Verify that the examples are running from both
> > >> ./bin/flink
> > >> > > > > and
> > >> > > > > > > > from
> > >> > > > > > > > > the web-based job submission tool(following items) -
> > >> SUCCESS
> > >> > > > > > > > > a. Start multiple task managers in the local cluster
> > >> > > > > > > > > b. Change the flink-conf.yml to define more than one
> > task
> > >> > > > > slot
> > >> > > > > > > (2)
> > >> > > > > > > > > c. Run the examples with a parallelism > 1
> > >> > > > > > > > > d. Examine the log output - no error messages should
> be
> > >> > > > > > > > encountered
> > >> > > > > > > > >
> > >> > > > > > > > > 4. Review the PR
> > >> > > > > > > > > - [Add 1.8 Release Blog Post] - Just a reminder,
> updated
> > >> the
> > >> > > > > > > release
> > >> > > > > > > > > date to correct date before merging.
> > >> > > > > > > > >
> > >> > > > > > > > > Cheers,
> > >> > > > > > > > > Jincheng
> > >> > > > > > > > >
> > >> > > > > > > > > Piotr Nowojski <[hidden email]> 于2019年3月25日周一
> > >> 下午4:11写道：
> > >> > > > > > > > >
> > >> > > > > > > > > > +1 from my side. Previously spotted performance
> > >> regression seems
> > >> > > > > to
> > >> > > > > > be
> > >> > > > > > > > > > gone, or mostly gone.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Piotrek
> > >> > > > > > > > > >
> > >> > > > > > > > > > > On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> > >> > > > [hidden email]>
> > >> > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > > Please review and vote on the release candidate 4
> > for
> > >> Flink
> > >> > > > > 1.8.0,
> > >> > > > > > as
> > >> > > > > > > > > > follows:
> > >> > > > > > > > > > > [ ] +1, Approve the release
> > >> > > > > > > > > > > [ ] -1, Do not approve the release (please provide
> > >> specific
> > >> > > > > > comments)
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > The complete staging area is available for your
> > >> review, which
> > >> > > > > > > includes:
> > >> > > > > > > > > > > * JIRA release notes [1],
> > >> > > > > > > > > > > * the official Apache source release and binary
> > >> convenience
> > >> > > > > > releases
> > >> > > > > > > to
> > >> > > > > > > > > > be deployed to dist.apache.org [2], which are
> signed
> > >> with the
> > >> > > > key
> > >> > > > > > > with
> > >> > > > > > > > > > fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
> > >> [3],
> > >> > > > > > > > > > > * all artifacts to be deployed to the Maven
> Central
> > >> Repository
> > >> > > > > [4],
> > >> > > > > > > > > > > * source code tag "release-1.8.0-rc4" [5],
> > >> > > > > > > > > > > * website pull request listing the new release [6]
> > >> > > > > > > > > > > * website pull request adding announcement blog
> post
> > >> [7].
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > The vote will be open for at least 72 hours. It is
> > >> adopted by
> > >> > > > > > > majority
> > >> > > > > > > > > > approval, with at least 3 PMC affirmative votes.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > Aljoscha
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [1]
> > >> > > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > >> > > > > > > > > > > [2]
> > >> > > > > https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > >> > > > > > > > > > > [3]
> > >> https://dist.apache.org/repos/dist/release/flink/KEYS
> > >> > > > > > > > > > > [4]
> > >> > > > > > > > > >
> > >> > > > > > >
> > >> > > >
> > >>
> https://repository.apache.org/content/repositories/orgapacheflink-1215
> > >> > > > > > > > > > > [5]
> > >> > > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > >> > > > > > > > > > > [6] https://github.com/apache/flink-web/pull/180
> <
> > >> > > > > > > > > > https://github.com/apache/flink-web/pull/180>
> > >> > > > > > > > > > > [7] https://github.com/apache/flink-web/pull/179
> <
> > >> > > > > > > > > > https://github.com/apache/flink-web/pull/179>
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > P.S. The difference to the previous RCs is small,
> > you
> > >> can fetch
> > >> > > > > the
> > >> > > > > > > > tags
> > >> > > > > > > > > > and do a "git log
> > release-1.8.0-rc1..release-1.8.0-rc4”
> > >> to see
> > >> > > > the
> > >> > > > > > > > > > difference in commits. Its fixes for the issues that
> > >> led to the
> > >> > > > > > > > > > cancellation of the previous RCs plus smaller fixes.
> > >> Most
> > >> > > > > > > > > > verification/testing that was carried out should
> apply
> > >> as is to
> > >> > > > > this
> > >> > > > > > > RC.
> > >> > > > > > > > > > Any functional verification that you did on previous
> > >> RCs should
> > >> > > > > > > > therefore
> > >> > > > > > > > > > easily carry over to this one.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> >
> > >>
> > >
> >
>

Aljoscha Krettek-2

Re: [VOTE] Release 1.8.0, release candidate #4

Thanks everyone, for discovering these!

@Richard Could you open a Jira issue for this and mark it as blocking?

> On 29. Mar 2019, at 09:22, Tzu-Li (Gordon) Tai <[hidden email]> wrote:
>
> @Yu discovered this issue, which IMO is probably a blocker for the
> release: https://issues.apache.org/jira/browse/FLINK-12064.
> The bug is a regression caused by previous state backend refactorings,
> which can result in incorrect representation of the schema of serialized
> keys in state because the wrong key serializer instance is being
> snapshotted.
>
> There is already a PR to fix this, I'll try to review and merge it over the
> weekend.
>
> On Fri, Mar 29, 2019 at 7:13 AM Richard Deurwaarder <[hidden email]> wrote:
>
>> -1 (non-binding)
>>
>> - Ran integration tests locally (1000+) of our flink job, all succeeded.
>> - Attempted to run job on hadoop, failed. It failed because we have a
>> firewall in place and we cannot set the rest port to a specific port/port
>> range.
>> Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
>> setting a REST port when running on yarn (
>>
>> https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
>> )
>> Code-wise it seems rather straightforward to fix but I am unsure about the
>> reason why this is hard-coded to 0 and what the impact would be.
>>
>> It would benefit us greatly if a fix for this could make it to 1.8.0.
>>
>> Regards,
>>
>> Richard
>>
>> On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <[hidden email]>
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Functional checks:
>>>
>>> - Built Flink from source (`mvn clean verify`) locally, with success
>>> - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
>>> (Hadoop 2.8.4, Scala 2.12)
>>> - Manually tested state schema evolution for POJO. Besides the tests that
>>> @Congxian already did, additionally tested evolution cases with POJO
>>> subclasses + non-registered POJOs.
>>> - Manually tested migration of Scala stateful jobs that use case classes
>> /
>>> Scala collections as state types, performing the migration across Scala
>>> 2.11 to Scala 2.12.
>>> - Reviewed release announcement PR
>>>
>>> Misc / legal checks:
>>>
>>> - checked checksums and signatures
>>> - No binaries in source distribution
>>> - Staging area does not seem to have any missing artifacts
>>>
>>> Cheers,
>>> Gordon
>>>
>>> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <[hidden email]
>>>
>>> wrote:
>>>
>>>> @Shaoxuan
>>>>
>>>> The drop in the serializerAvro benchmark, as explained earlier in
>>> previous
>>>> voting threads of earlier RCs, was due to a slower job initialization
>>> phase
>>>> caused by slower deserialization of the AvroSerializer.
>>>> Piotr also pointed out that after the number of records was increased
>> in
>>>> the serializer benchmarks, this drop was no longer observable before /
>>>> after the changes in mid February.
>>>> IMO, this is not critical as it does not affect the per-record
>>> performance
>>>> / throughput, and therefore should not block this release.
>>>>
>>>> On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <
>> [hidden email]>
>>>> wrote:
>>>>
>>>>> By now, I'm reasonably sure that the test instabilities on the
>>> end-to-end
>>>>> test are only instabilities. I pushed changes to increase timeouts to
>>> make
>>>>> the tests more stable. As in any project, there will always be bugs
>> but
>>> I
>>>>> think we could release this RC4 and be reasonably sure that it works
>>> well.
>>>>>
>>>>> Now, we only need to have the required number of PMC votes.
>>>>>
>>>>> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
>>>>>> +1 （non-binding)
>>>>>>
>>>>>> • checked signature and checksum ok
>>>>>> • mvn clean package -DskipTests ok
>>>>>> • Run job on yarn ok
>>>>>> • Test state migration with POJO type (both heap and rocksdb) ok
>>>>>> • - 1.6 -> 1.8
>>>>>> • - 1.7 -> 1.8
>>>>>> • - 1.8 -> 1.8
>>>>>>
>>>>>>
>>>>>> Best, Congxian
>>>>>> On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>,
>>> wrote:
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> - checked JIRA release note
>>>>>>> - ran "mvn package -DskipTests"
>>>>>>> - checked signature and checksum
>>>>>>> - started a cluster locally and ran some examples in binary
>>>>>>> - checked web site announcement's PR
>>>>>>>
>>>>>>> Best,
>>>>>>> Vino
>>>>>>>
>>>>>>>
>>>>>>> Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
>>>>>>>
>>>>>>>> +1 (non-binding)
>>>>>>>>
>>>>>>>> - checked checksums and GPG files
>>>>>>>> - build from source successfully- run end-to-end precommit tests
>>>>>>>> successfully- run end-to-end nightly tests successfully
>>>>>>>> Xiaowei
>>>>>>>> On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
>>>>> [hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> +1 (non-binding)
>>>>>>>>
>>>>>>>> - Checked release notes: OK
>>>>>>>> - Checked sums and signatures: OK
>>>>>>>> - Source release
>>>>>>>> - contains no binaries: OK
>>>>>>>> - contains no 1.8-SNAPSHOT references: OK
>>>>>>>> - build from source: OK (8u101)
>>>>>>>> - mvn clean verify: OK (8u101)
>>>>>>>> - Binary release
>>>>>>>> - no examples appear to be missing
>>>>>>>> - started a cluster; WebUI reachable, example ran successfully
>>>>>>>> - end-to-end test (all but K8S and docker ones): OK (8u101)
>>>>>>>> - Repository appears to contain all expected artifacts
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Yu
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]>
>>> wrote:
>>>>>>>>
>>>>>>>>> +1 (non-binding)
>>>>>>>>>
>>>>>>>>> Checked items:
>>>>>>>>> - checked checksums and GPG files
>>>>>>>>> - verified that the source archives do not contains any
>> binaries
>>>>>>>>> - checked that all POM files point to the same version
>>>>>>>>> - build from source successfully
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Kurt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
>>>>> [hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 (non-binding)
>>>>>>>>>>
>>>>>>>>>> I tested RC4 with the following items:
>>>>>>>>>> - Maven Central Repository contains all artifacts
>>>>>>>>>> - Built the source with Maven (ensured all source files have
>>>>> Apache
>>>>>>>>>> headers), and executed built-in tests via "mvn clean verify"
>>>>>>>>>> - Manually executed the tests in IntelliJ IDE
>>>>>>>>>> - Verify that the quickstarts for Scala and Java are working
>>>>> with the
>>>>>>>>>> staging repository in IntelliJ
>>>>>>>>>> - Checked the benchmark results. The perf regression of
>>>>>>>>>> tuple-key-by/statebackend/tumblingWindow are gone, but the
>>>>> regression
>>>>>>>> on
>>>>>>>>>> serializer still exists.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Shaoxuan
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
>>>>> [hidden email]
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Aljoscha, I think you are right, increase the timeout
>>>>> config will
>>>>>>>>> fix
>>>>>>>>>>> this issue. this depends on the resource of Travis. I
>> would
>>>>> like
>>>>>>>> share
>>>>>>>>>>> some phenomenon during my test (not the flink problem) as
>>>>> follows:
>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>> During my testing, `mvn clean verify` and `nightly
>>> end-to-end
>>>>> test `
>>>>>>>>> both
>>>>>>>>>>> consume a lot of machine resources (especially
>>>>> memory/network), and
>>>>>>>> the
>>>>>>>>>>> network bandwidth requirements of `nightly end-to-end
>> test `
>>>>> are also
>>>>>>>>>> very
>>>>>>>>>>> high. In China, need to use VPN acceleration (100~200Kb
>>> before
>>>>>>>>>>> acceleration, 3~4Mb after acceleration), I have
>> encountered:
>>>>> [Avro
>>>>>>>>>>> Confluent Schema Registry nightly end-to-end test' failed
>>>>> after 18
>>>>>>>>>> minutes
>>>>>>>>>>> and 15 seconds! Test exited with exit Code 1] takes more
>>> than
>>>>> 18
>>>>>>>>> minutes,
>>>>>>>>>>> the download failed because the network bandwidth is not
>>>>> enough. and
>>>>>>>> it
>>>>>>>>>>> runs smoothly when using VPN acceleration. The overall
>>>>> end-to-end run
>>>>>>>>> was
>>>>>>>>>>> passed twice. The Docker resource configuration (CUPs 7,
>>> Mem:
>>>>> 28.7G,
>>>>>>>>>> Swap:
>>>>>>>>>>> 3.5G). See detail log here
>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> Just now, I had checked the Travis for your last commit
>>>>> (Increase
>>>>>>>>> startup
>>>>>>>>>>> timeout in end-to-end tests), in addition to the Cleanup
>>>>> phase, other
>>>>>>>>>>> phases are successful. here
>>>>>>>>>>> <https://travis-ci.org/apache/flink/builds/511071777>
>>>>>>>>>>>
>>>>>>>>>>> In order to verify that our speculation is accurate, I can
>>>>> help with
>>>>>>>> 10
>>>>>>>>>> and
>>>>>>>>>>> 20 seconds timeout config on my repo verification to see
>> if
>>>>> 100%
>>>>>>>>>> recurring
>>>>>>>>>>> timeout problem. It is already running, we are waiting for
>>> the
>>>>>>>> result.
>>>>>>>>>>> 10seconds <
>>>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235749
>>>>>>>>>>
>>>>>>>>>>> 20seconds <
>>>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235598
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Jincheng
>>>>>>>>>>>
>>>>>>>>>>> Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
>>>>> 上午1:04写道：
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the testing done so far!
>>>>>>>>>>>>
>>>>>>>>>>>> There has been quite some flakiness on Travis lately,
>> see
>>>>> here:
>>>>>>>>>>>> https://travis-ci.org/apache/flink/branches <
>>>>>>>>>>>> https://travis-ci.org/apache/flink/branches>. I’m a bit
>>>>> hesitant
>>>>>>>> to
>>>>>>>>>>>> release in this state. Looking at the tests you can see
>>>>> that all of
>>>>>>>>> the
>>>>>>>>>>>> end-to-end tests fail because waiting for the dispatcher
>>> to
>>>>> come up
>>>>>>>>>> times
>>>>>>>>>>>> out. I also noticed that this usually takes about 5-8
>>>>> seconds on
>>>>>>>>>> Travis,
>>>>>>>>>>> so
>>>>>>>>>>>> a 10 second timeout might be a bit low. I pushed commits
>>> to
>>>>>>>> increase
>>>>>>>>>> that
>>>>>>>>>>>> to 20 secs. Let’s see what will happen.
>>>>>>>>>>>>
>>>>>>>>>>>> I’ll keep you posted!
>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>
>>>>>>>>>>>>> On 25. Mar 2019, at 13:13, jincheng sun <
>>>>>>>> [hidden email]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Great thanks for preparing the RC4 of Flink 1.8.0,
>>>>> Aljoscha!
>>>>>>>>>>>>>
>>>>>>>>>>>>> +1 (non-binding)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I checked the functional things as follows(Without
>>>>> performance
>>>>>>>>>>>>> verification):
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Checking Artifacts：
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1). Download the release source code - SUCCESS
>>>>>>>>>>>>> 2). Check Source release flink-1.8.0-src.tgz.sha512 -
>>>>> SUCCESS
>>>>>>>>>>>>> 3). Download the released JAR - SUCCESS
>>>>>>>>>>>>> 4). Check if checksums and GPG files match the
>>>>> corresponding
>>>>>>>>>> release
>>>>>>>>>>>>> files - SUCCESS.
>>>>>>>>>>>>> 5). Verify that the source archives do not contain any
>>>>>>>> binaries
>>>>>>>>> -
>>>>>>>>>>>>> SUCCESS.
>>>>>>>>>>>>> 6). Build the source with `mvn clean verify
>> -DskipTests`
>>>>> to
>>>>>>>>> ensure
>>>>>>>>>>> all
>>>>>>>>>>>>> source files have Apache headers - SUCCESS
>>>>>>>>>>>>> 7). Check that all POM files point to the same
>> version -
>>>>>>>> SUCCESS
>>>>>>>>>>>>> 8). Read the `README.md` file to ensure there is
>> nothing
>>>>>>>>>> unexpected
>>>>>>>>>>> -
>>>>>>>>>>>>> SUCCESS
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Testing Larger Setups
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cluster Environment：7 nodes, jm 1024m, tm 4096m
>>>>>>>>>>>>> Testing Jobs: WordCount(Batch&Streaming),
>>>>>>>>>>> DataStreamAllroundTestProgram
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1). Use local&hdfs file systems for checkpoints -
>>> SUCCESS
>>>>>>>>>>>>> 2). Use hdfs file systems for input/output -SUCCESS
>>>>>>>>>>>>> 3). Run examples on YARN(with or without session) -
>>>>> SUCCESS
>>>>>>>>>>>>> 4). Test failover and recovery. - SUCCESS
>>>>>>>>>>>>> 5). Test incremental&non-incremental checkpoint -
>>> SUCCESS
>>>>>>>>>>>>> 6). Test connector - kafka -SUCCESS
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3. Testing Functionality
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1). Built-in tests(linux&mac os)
>>>>>>>>>>>>> - `mvn cealn verify` (some test timeout error and test
>>>>> case
>>>>>>>>> bug
>>>>>>>>>>> see
>>>>>>>>>>>>> FLINK-12001 <
>>>>> https://issues.apache.org/jira/browse/FLINK-12001>,
>>>>>>>>> all
>>>>>>>>>>> of
>>>>>>>>>>>>> them are not the blocker)
>>>>>>>>>>>>> - build for scala 2.11（mvn clean install -P scala-2.11
>>>>>>>>>>> -DskipTests）
>>>>>>>>>>>>> - SUCCESS
>>>>>>>>>>>>> - Run the scripted nightly end-to-end test - SUCCESS
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2). Quickstarts
>>>>>>>>>>>>> - Verify that the quickstarts for Scala with the
>> staging
>>>>>>>>>>> repository
>>>>>>>>>>>>> in IntelliJ - SUCCESS
>>>>>>>>>>>>> - Verify that the quickstarts for Java with the
>> staging
>>>>>>>>>> repository
>>>>>>>>>>>> in
>>>>>>>>>>>>> IntelliJ - SUCCESS
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3). Simple Starter Experience and Use Cases
>>>>>>>>>>>>>
>>>>>>>>>>>>> - run all examples from IntelliJ IDE - SUCCESS
>>>>>>>>>>>>> - Start a local cluster and verify that the processes
>> -
>>>>>>>>> SUCCESS
>>>>>>>>>>>>> a. Examine the *.out files (should be empty) and the
>> log
>>>>>>>>> files
>>>>>>>>>>>>> (should contain no exceptions)
>>>>>>>>>>>>> b. Test for Linux, MacOS
>>>>>>>>>>>>> c. Shutdown and verify there are no exceptions in the
>>> log
>>>>>>>>>> output
>>>>>>>>>>>>> (after shutdown)
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Verify that the examples are running from both
>>>>> ./bin/flink
>>>>>>>>> and
>>>>>>>>>>>> from
>>>>>>>>>>>>> the web-based job submission tool(following items) -
>>>>> SUCCESS
>>>>>>>>>>>>> a. Start multiple task managers in the local cluster
>>>>>>>>>>>>> b. Change the flink-conf.yml to define more than one
>>> task
>>>>>>>>> slot
>>>>>>>>>>> (2)
>>>>>>>>>>>>> c. Run the examples with a parallelism > 1
>>>>>>>>>>>>> d. Examine the log output - no error messages should
>> be
>>>>>>>>>>>> encountered
>>>>>>>>>>>>>
>>>>>>>>>>>>> 4. Review the PR
>>>>>>>>>>>>> - [Add 1.8 Release Blog Post] - Just a reminder,
>> updated
>>>>> the
>>>>>>>>>>> release
>>>>>>>>>>>>> date to correct date before merging.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Jincheng
>>>>>>>>>>>>>
>>>>>>>>>>>>> Piotr Nowojski <[hidden email]> 于2019年3月25日周一
>>>>> 下午4:11写道：
>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1 from my side. Previously spotted performance
>>>>> regression seems
>>>>>>>>> to
>>>>>>>>>> be
>>>>>>>>>>>>>> gone, or mostly gone.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Piotrek
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <
>>>>>>>> [hidden email]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>> Please review and vote on the release candidate 4
>>> for
>>>>> Flink
>>>>>>>>> 1.8.0,
>>>>>>>>>> as
>>>>>>>>>>>>>> follows:
>>>>>>>>>>>>>>> [ ] +1, Approve the release
>>>>>>>>>>>>>>> [ ] -1, Do not approve the release (please provide
>>>>> specific
>>>>>>>>>> comments)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The complete staging area is available for your
>>>>> review, which
>>>>>>>>>>> includes:
>>>>>>>>>>>>>>> * JIRA release notes [1],
>>>>>>>>>>>>>>> * the official Apache source release and binary
>>>>> convenience
>>>>>>>>>> releases
>>>>>>>>>>> to
>>>>>>>>>>>>>> be deployed to dist.apache.org [2], which are
>> signed
>>>>> with the
>>>>>>>> key
>>>>>>>>>>> with
>>>>>>>>>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
>>>>> [3],
>>>>>>>>>>>>>>> * all artifacts to be deployed to the Maven
>> Central
>>>>> Repository
>>>>>>>>> [4],
>>>>>>>>>>>>>>> * source code tag "release-1.8.0-rc4" [5],
>>>>>>>>>>>>>>> * website pull request listing the new release [6]
>>>>>>>>>>>>>>> * website pull request adding announcement blog
>> post
>>>>> [7].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The vote will be open for at least 72 hours. It is
>>>>> adopted by
>>>>>>>>>>> majority
>>>>>>>>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
>>>>>>>>>>>>>>> [2]
>>>>>>>>> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
>>>>>>>>>>>>>>> [3]
>>>>> https://dist.apache.org/repos/dist/release/flink/KEYS
>>>>>>>>>>>>>>> [4]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>
>> https://repository.apache.org/content/repositories/orgapacheflink-1215
>>>>>>>>>>>>>>> [5]
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
>>>>>>>>>>>>>>> [6] https://github.com/apache/flink-web/pull/180
>> <
>>>>>>>>>>>>>> https://github.com/apache/flink-web/pull/180>
>>>>>>>>>>>>>>> [7] https://github.com/apache/flink-web/pull/179
>> <
>>>>>>>>>>>>>> https://github.com/apache/flink-web/pull/179>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> P.S. The difference to the previous RCs is small,
>>> you
>>>>> can fetch
>>>>>>>>> the
>>>>>>>>>>>> tags
>>>>>>>>>>>>>> and do a "git log
>>> release-1.8.0-rc1..release-1.8.0-rc4”
>>>>> to see
>>>>>>>> the
>>>>>>>>>>>>>> difference in commits. Its fixes for the issues that
>>>>> led to the
>>>>>>>>>>>>>> cancellation of the previous RCs plus smaller fixes.
>>>>> Most
>>>>>>>>>>>>>> verification/testing that was carried out should
>> apply
>>>>> as is to
>>>>>>>>> this
>>>>>>>>>>> RC.
>>>>>>>>>>>>>> Any functional verification that you did on previous
>>>>> RCs should
>>>>>>>>>>>> therefore
>>>>>>>>>>>>>> easily carry over to this one.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Aljoscha Krettek-2

Re: [VOTE] Release 1.8.0, release candidate #4

In reply to this post by Richard Deurwaarder

@Richard Did this work for you previously? From the change, it seems that the port was always set to 0 on YARN even before.

> On 28. Mar 2019, at 16:13, Richard Deurwaarder <[hidden email]> wrote:
>
> -1 (non-binding)
>
> - Ran integration tests locally (1000+) of our flink job, all succeeded.
> - Attempted to run job on hadoop, failed. It failed because we have a
> firewall in place and we cannot set the rest port to a specific port/port
> range.
> Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
> setting a REST port when running on yarn (
> https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
> )
> Code-wise it seems rather straightforward to fix but I am unsure about the
> reason why this is hard-coded to 0 and what the impact would be.
>
> It would benefit us greatly if a fix for this could make it to 1.8.0.
>
> Regards,
>
> Richard
>
> On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <[hidden email]>
> wrote:
>
>> +1 (binding)
>>
>> Functional checks:
>>
>> - Built Flink from source (`mvn clean verify`) locally, with success
>> - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
>> (Hadoop 2.8.4, Scala 2.12)
>> - Manually tested state schema evolution for POJO. Besides the tests that
>> @Congxian already did, additionally tested evolution cases with POJO
>> subclasses + non-registered POJOs.
>> - Manually tested migration of Scala stateful jobs that use case classes /
>> Scala collections as state types, performing the migration across Scala
>> 2.11 to Scala 2.12.
>> - Reviewed release announcement PR
>>
>> Misc / legal checks:
>>
>> - checked checksums and signatures
>> - No binaries in source distribution
>> - Staging area does not seem to have any missing artifacts
>>
>> Cheers,
>> Gordon
>>
>> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <[hidden email]>
>> wrote:
>>
>>> @Shaoxuan
>>>
>>> The drop in the serializerAvro benchmark, as explained earlier in
>> previous
>>> voting threads of earlier RCs, was due to a slower job initialization
>> phase
>>> caused by slower deserialization of the AvroSerializer.
>>> Piotr also pointed out that after the number of records was increased in
>>> the serializer benchmarks, this drop was no longer observable before /
>>> after the changes in mid February.
>>> IMO, this is not critical as it does not affect the per-record
>> performance
>>> / throughput, and therefore should not block this release.
>>>
>>> On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <[hidden email]>
>>> wrote:
>>>
>>>> By now, I'm reasonably sure that the test instabilities on the
>> end-to-end
>>>> test are only instabilities. I pushed changes to increase timeouts to
>> make
>>>> the tests more stable. As in any project, there will always be bugs but
>> I
>>>> think we could release this RC4 and be reasonably sure that it works
>> well.
>>>>
>>>> Now, we only need to have the required number of PMC votes.
>>>>
>>>> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
>>>>> +1 （non-binding)
>>>>>
>>>>> • checked signature and checksum ok
>>>>> • mvn clean package -DskipTests ok
>>>>> • Run job on yarn ok
>>>>> • Test state migration with POJO type (both heap and rocksdb) ok
>>>>> • - 1.6 -> 1.8
>>>>> • - 1.7 -> 1.8
>>>>> • - 1.8 -> 1.8
>>>>>
>>>>>
>>>>> Best, Congxian
>>>>> On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>,
>> wrote:
>>>>>> +1 (non-binding)
>>>>>>
>>>>>> - checked JIRA release note
>>>>>> - ran "mvn package -DskipTests"
>>>>>> - checked signature and checksum
>>>>>> - started a cluster locally and ran some examples in binary
>>>>>> - checked web site announcement's PR
>>>>>>
>>>>>> Best,
>>>>>> Vino
>>>>>>
>>>>>>
>>>>>> Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> - checked checksums and GPG files
>>>>>>> - build from source successfully- run end-to-end precommit tests
>>>>>>> successfully- run end-to-end nightly tests successfully
>>>>>>> Xiaowei
>>>>>>> On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
>>>> [hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>> +1 (non-binding)
>>>>>>>
>>>>>>> - Checked release notes: OK
>>>>>>> - Checked sums and signatures: OK
>>>>>>> - Source release
>>>>>>> - contains no binaries: OK
>>>>>>> - contains no 1.8-SNAPSHOT references: OK
>>>>>>> - build from source: OK (8u101)
>>>>>>> - mvn clean verify: OK (8u101)
>>>>>>> - Binary release
>>>>>>> - no examples appear to be missing
>>>>>>> - started a cluster; WebUI reachable, example ran successfully
>>>>>>> - end-to-end test (all but K8S and docker ones): OK (8u101)
>>>>>>> - Repository appears to contain all expected artifacts
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Yu
>>>>>>>
>>>>>>>
>>>>>>> On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]>
>> wrote:
>>>>>>>
>>>>>>>> +1 (non-binding)
>>>>>>>>
>>>>>>>> Checked items:
>>>>>>>> - checked checksums and GPG files
>>>>>>>> - verified that the source archives do not contains any binaries
>>>>>>>> - checked that all POM files point to the same version
>>>>>>>> - build from source successfully
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Kurt
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
>>>> [hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 (non-binding)
>>>>>>>>>
>>>>>>>>> I tested RC4 with the following items:
>>>>>>>>> - Maven Central Repository contains all artifacts
>>>>>>>>> - Built the source with Maven (ensured all source files have
>>>> Apache
>>>>>>>>> headers), and executed built-in tests via "mvn clean verify"
>>>>>>>>> - Manually executed the tests in IntelliJ IDE
>>>>>>>>> - Verify that the quickstarts for Scala and Java are working
>>>> with the
>>>>>>>>> staging repository in IntelliJ
>>>>>>>>> - Checked the benchmark results. The perf regression of
>>>>>>>>> tuple-key-by/statebackend/tumblingWindow are gone, but the
>>>> regression
>>>>>>> on
>>>>>>>>> serializer still exists.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Shaoxuan
>>>>>>>>>
>>>>>>>>> On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
>>>> [hidden email]
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Aljoscha, I think you are right, increase the timeout
>>>> config will
>>>>>>>> fix
>>>>>>>>>> this issue. this depends on the resource of Travis. I would
>>>> like
>>>>>>> share
>>>>>>>>>> some phenomenon during my test (not the flink problem) as
>>>> follows:
>>>>>>> :-)
>>>>>>>>>>
>>>>>>>>>> During my testing, `mvn clean verify` and `nightly
>> end-to-end
>>>> test `
>>>>>>>> both
>>>>>>>>>> consume a lot of machine resources (especially
>>>> memory/network), and
>>>>>>> the
>>>>>>>>>> network bandwidth requirements of `nightly end-to-end test `
>>>> are also
>>>>>>>>> very
>>>>>>>>>> high. In China, need to use VPN acceleration (100~200Kb
>> before
>>>>>>>>>> acceleration, 3~4Mb after acceleration), I have encountered:
>>>> [Avro
>>>>>>>>>> Confluent Schema Registry nightly end-to-end test' failed
>>>> after 18
>>>>>>>>> minutes
>>>>>>>>>> and 15 seconds! Test exited with exit Code 1] takes more
>> than
>>>> 18
>>>>>>>> minutes,
>>>>>>>>>> the download failed because the network bandwidth is not
>>>> enough. and
>>>>>>> it
>>>>>>>>>> runs smoothly when using VPN acceleration. The overall
>>>> end-to-end run
>>>>>>>> was
>>>>>>>>>> passed twice. The Docker resource configuration (CUPs 7,
>> Mem:
>>>> 28.7G,
>>>>>>>>> Swap:
>>>>>>>>>> 3.5G). See detail log here
>>>>>>>>>> <
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
>>>>>>>>>>>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>> Just now, I had checked the Travis for your last commit
>>>> (Increase
>>>>>>>> startup
>>>>>>>>>> timeout in end-to-end tests), in addition to the Cleanup
>>>> phase, other
>>>>>>>>>> phases are successful. here
>>>>>>>>>> <https://travis-ci.org/apache/flink/builds/511071777>
>>>>>>>>>>
>>>>>>>>>> In order to verify that our speculation is accurate, I can
>>>> help with
>>>>>>> 10
>>>>>>>>> and
>>>>>>>>>> 20 seconds timeout config on my repo verification to see if
>>>> 100%
>>>>>>>>> recurring
>>>>>>>>>> timeout problem. It is already running, we are waiting for
>> the
>>>>>>> result.
>>>>>>>>>> 10seconds <
>>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235749
>>>>>>>>>
>>>>>>>>>> 20seconds <
>>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235598
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Jincheng
>>>>>>>>>>
>>>>>>>>>> Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
>>>> 上午1:04写道：
>>>>>>>>>>
>>>>>>>>>>> Thanks for the testing done so far!
>>>>>>>>>>>
>>>>>>>>>>> There has been quite some flakiness on Travis lately, see
>>>> here:
>>>>>>>>>>> https://travis-ci.org/apache/flink/branches <
>>>>>>>>>>> https://travis-ci.org/apache/flink/branches>. I’m a bit
>>>> hesitant
>>>>>>> to
>>>>>>>>>>> release in this state. Looking at the tests you can see
>>>> that all of
>>>>>>>> the
>>>>>>>>>>> end-to-end tests fail because waiting for the dispatcher
>> to
>>>> come up
>>>>>>>>> times
>>>>>>>>>>> out. I also noticed that this usually takes about 5-8
>>>> seconds on
>>>>>>>>> Travis,
>>>>>>>>>> so
>>>>>>>>>>> a 10 second timeout might be a bit low. I pushed commits
>> to
>>>>>>> increase
>>>>>>>>> that
>>>>>>>>>>> to 20 secs. Let’s see what will happen.
>>>>>>>>>>>
>>>>>>>>>>> I’ll keep you posted!
>>>>>>>>>>> Aljoscha
>>>>>>>>>>>
>>>>>>>>>>>> On 25. Mar 2019, at 13:13, jincheng sun <
>>>>>>> [hidden email]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Great thanks for preparing the RC4 of Flink 1.8.0,
>>>> Aljoscha!
>>>>>>>>>>>>
>>>>>>>>>>>> +1 (non-binding)
>>>>>>>>>>>>
>>>>>>>>>>>> I checked the functional things as follows(Without
>>>> performance
>>>>>>>>>>>> verification):
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Checking Artifacts：
>>>>>>>>>>>>
>>>>>>>>>>>> 1). Download the release source code - SUCCESS
>>>>>>>>>>>> 2). Check Source release flink-1.8.0-src.tgz.sha512 -
>>>> SUCCESS
>>>>>>>>>>>> 3). Download the released JAR - SUCCESS
>>>>>>>>>>>> 4). Check if checksums and GPG files match the
>>>> corresponding
>>>>>>>>> release
>>>>>>>>>>>> files - SUCCESS.
>>>>>>>>>>>> 5). Verify that the source archives do not contain any
>>>>>>> binaries
>>>>>>>> -
>>>>>>>>>>>> SUCCESS.
>>>>>>>>>>>> 6). Build the source with `mvn clean verify -DskipTests`
>>>> to
>>>>>>>> ensure
>>>>>>>>>> all
>>>>>>>>>>>> source files have Apache headers - SUCCESS
>>>>>>>>>>>> 7). Check that all POM files point to the same version -
>>>>>>> SUCCESS
>>>>>>>>>>>> 8). Read the `README.md` file to ensure there is nothing
>>>>>>>>> unexpected
>>>>>>>>>> -
>>>>>>>>>>>> SUCCESS
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Testing Larger Setups
>>>>>>>>>>>>
>>>>>>>>>>>> Cluster Environment：7 nodes, jm 1024m, tm 4096m
>>>>>>>>>>>> Testing Jobs: WordCount(Batch&Streaming),
>>>>>>>>>> DataStreamAllroundTestProgram
>>>>>>>>>>>>
>>>>>>>>>>>> 1). Use local&hdfs file systems for checkpoints -
>> SUCCESS
>>>>>>>>>>>> 2). Use hdfs file systems for input/output -SUCCESS
>>>>>>>>>>>> 3). Run examples on YARN(with or without session) -
>>>> SUCCESS
>>>>>>>>>>>> 4). Test failover and recovery. - SUCCESS
>>>>>>>>>>>> 5). Test incremental&non-incremental checkpoint -
>> SUCCESS
>>>>>>>>>>>> 6). Test connector - kafka -SUCCESS
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Testing Functionality
>>>>>>>>>>>>
>>>>>>>>>>>> 1). Built-in tests(linux&mac os)
>>>>>>>>>>>> - `mvn cealn verify` (some test timeout error and test
>>>> case
>>>>>>>> bug
>>>>>>>>>> see
>>>>>>>>>>>> FLINK-12001 <
>>>> https://issues.apache.org/jira/browse/FLINK-12001>,
>>>>>>>> all
>>>>>>>>>> of
>>>>>>>>>>>> them are not the blocker)
>>>>>>>>>>>> - build for scala 2.11（mvn clean install -P scala-2.11
>>>>>>>>>> -DskipTests）
>>>>>>>>>>>> - SUCCESS
>>>>>>>>>>>> - Run the scripted nightly end-to-end test - SUCCESS
>>>>>>>>>>>>
>>>>>>>>>>>> 2). Quickstarts
>>>>>>>>>>>> - Verify that the quickstarts for Scala with the staging
>>>>>>>>>> repository
>>>>>>>>>>>> in IntelliJ - SUCCESS
>>>>>>>>>>>> - Verify that the quickstarts for Java with the staging
>>>>>>>>> repository
>>>>>>>>>>> in
>>>>>>>>>>>> IntelliJ - SUCCESS
>>>>>>>>>>>>
>>>>>>>>>>>> 3). Simple Starter Experience and Use Cases
>>>>>>>>>>>>
>>>>>>>>>>>> - run all examples from IntelliJ IDE - SUCCESS
>>>>>>>>>>>> - Start a local cluster and verify that the processes -
>>>>>>>> SUCCESS
>>>>>>>>>>>> a. Examine the *.out files (should be empty) and the log
>>>>>>>> files
>>>>>>>>>>>> (should contain no exceptions)
>>>>>>>>>>>> b. Test for Linux, MacOS
>>>>>>>>>>>> c. Shutdown and verify there are no exceptions in the
>> log
>>>>>>>>> output
>>>>>>>>>>>> (after shutdown)
>>>>>>>>>>>>
>>>>>>>>>>>> - Verify that the examples are running from both
>>>> ./bin/flink
>>>>>>>> and
>>>>>>>>>>> from
>>>>>>>>>>>> the web-based job submission tool(following items) -
>>>> SUCCESS
>>>>>>>>>>>> a. Start multiple task managers in the local cluster
>>>>>>>>>>>> b. Change the flink-conf.yml to define more than one
>> task
>>>>>>>> slot
>>>>>>>>>> (2)
>>>>>>>>>>>> c. Run the examples with a parallelism > 1
>>>>>>>>>>>> d. Examine the log output - no error messages should be
>>>>>>>>>>> encountered
>>>>>>>>>>>>
>>>>>>>>>>>> 4. Review the PR
>>>>>>>>>>>> - [Add 1.8 Release Blog Post] - Just a reminder, updated
>>>> the
>>>>>>>>>> release
>>>>>>>>>>>> date to correct date before merging.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Jincheng
>>>>>>>>>>>>
>>>>>>>>>>>> Piotr Nowojski <[hidden email]> 于2019年3月25日周一
>>>> 下午4:11写道：
>>>>>>>>>>>>
>>>>>>>>>>>>> +1 from my side. Previously spotted performance
>>>> regression seems
>>>>>>>> to
>>>>>>>>> be
>>>>>>>>>>>>> gone, or mostly gone.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Piotrek
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <
>>>>>>> [hidden email]>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>> Please review and vote on the release candidate 4
>> for
>>>> Flink
>>>>>>>> 1.8.0,
>>>>>>>>> as
>>>>>>>>>>>>> follows:
>>>>>>>>>>>>>> [ ] +1, Approve the release
>>>>>>>>>>>>>> [ ] -1, Do not approve the release (please provide
>>>> specific
>>>>>>>>> comments)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The complete staging area is available for your
>>>> review, which
>>>>>>>>>> includes:
>>>>>>>>>>>>>> * JIRA release notes [1],
>>>>>>>>>>>>>> * the official Apache source release and binary
>>>> convenience
>>>>>>>>> releases
>>>>>>>>>> to
>>>>>>>>>>>>> be deployed to dist.apache.org [2], which are signed
>>>> with the
>>>>>>> key
>>>>>>>>>> with
>>>>>>>>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
>>>> [3],
>>>>>>>>>>>>>> * all artifacts to be deployed to the Maven Central
>>>> Repository
>>>>>>>> [4],
>>>>>>>>>>>>>> * source code tag "release-1.8.0-rc4" [5],
>>>>>>>>>>>>>> * website pull request listing the new release [6]
>>>>>>>>>>>>>> * website pull request adding announcement blog post
>>>> [7].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The vote will be open for at least 72 hours. It is
>>>> adopted by
>>>>>>>>>> majority
>>>>>>>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Aljoscha
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
>>>>>>>>>>>>>> [2]
>>>>>>>> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
>>>>>>>>>>>>>> [3]
>>>> https://dist.apache.org/repos/dist/release/flink/KEYS
>>>>>>>>>>>>>> [4]
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>> https://repository.apache.org/content/repositories/orgapacheflink-1215
>>>>>>>>>>>>>> [5]
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
>>>>>>>>>>>>>> [6] https://github.com/apache/flink-web/pull/180 <
>>>>>>>>>>>>> https://github.com/apache/flink-web/pull/180>
>>>>>>>>>>>>>> [7] https://github.com/apache/flink-web/pull/179 <
>>>>>>>>>>>>> https://github.com/apache/flink-web/pull/179>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> P.S. The difference to the previous RCs is small,
>> you
>>>> can fetch
>>>>>>>> the
>>>>>>>>>>> tags
>>>>>>>>>>>>> and do a "git log
>> release-1.8.0-rc1..release-1.8.0-rc4”
>>>> to see
>>>>>>> the
>>>>>>>>>>>>> difference in commits. Its fixes for the issues that
>>>> led to the
>>>>>>>>>>>>> cancellation of the previous RCs plus smaller fixes.
>>>> Most
>>>>>>>>>>>>> verification/testing that was carried out should apply
>>>> as is to
>>>>>>>> this
>>>>>>>>>> RC.
>>>>>>>>>>>>> Any functional verification that you did on previous
>>>> RCs should
>>>>>>>>>>> therefore
>>>>>>>>>>>>> easily carry over to this one.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>
>>>
>>

Rong Rong

Re: [VOTE] Release 1.8.0, release candidate #4

Hi @Aljoscha,

Based on the previous commit [1] that adds the random port selection code,
it seems like the important part is to unset whatever 'rest.port' setting
previously done. I don't think the current way of setting the BIND_PORT
actually overrides any existing PORT setting. However, I wasn't able to
find any test that is related, maybe @Till can provide more insight here?

Maybe @Richard can provide more detail on the YARN run command used to
reproduce the problem?

Thanks,
Rong

[1]
https://github.com/apache/flink/commit/dbe0e8286d76a5facdb49589b638b87dbde80178#diff-487838863ab693af7008f04cb3359be3R117

On Sat, Mar 30, 2019 at 5:51 AM Aljoscha Krettek <[hidden email]>
wrote:

> @Richard Did this work for you previously? From the change, it seems that
> the port was always set to 0 on YARN even before.
>
> > On 28. Mar 2019, at 16:13, Richard Deurwaarder <[hidden email]> wrote:
> >
> > -1 (non-binding)
> >
> > - Ran integration tests locally (1000+) of our flink job, all succeeded.
> > - Attempted to run job on hadoop, failed. It failed because we have a
> > firewall in place and we cannot set the rest port to a specific port/port
> > range.
> > Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
> > setting a REST port when running on yarn (
> >
> https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
> > )
> > Code-wise it seems rather straightforward to fix but I am unsure about
> the
> > reason why this is hard-coded to 0 and what the impact would be.
> >
> > It would benefit us greatly if a fix for this could make it to 1.8.0.
> >
> > Regards,
> >
> > Richard
> >
> > On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <[hidden email]
> >
> > wrote:
> >
> >> +1 (binding)
> >>
> >> Functional checks:
> >>
> >> - Built Flink from source (`mvn clean verify`) locally, with success
> >> - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
> >> (Hadoop 2.8.4, Scala 2.12)
> >> - Manually tested state schema evolution for POJO. Besides the tests
> that
> >> @Congxian already did, additionally tested evolution cases with POJO
> >> subclasses + non-registered POJOs.
> >> - Manually tested migration of Scala stateful jobs that use case
> classes /
> >> Scala collections as state types, performing the migration across Scala
> >> 2.11 to Scala 2.12.
> >> - Reviewed release announcement PR
> >>
> >> Misc / legal checks:
> >>
> >> - checked checksums and signatures
> >> - No binaries in source distribution
> >> - Staging area does not seem to have any missing artifacts
> >>
> >> Cheers,
> >> Gordon
> >>
> >> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <
> [hidden email]>
> >> wrote:
> >>
> >>> @Shaoxuan
> >>>
> >>> The drop in the serializerAvro benchmark, as explained earlier in
> >> previous
> >>> voting threads of earlier RCs, was due to a slower job initialization
> >> phase
> >>> caused by slower deserialization of the AvroSerializer.
> >>> Piotr also pointed out that after the number of records was increased
> in
> >>> the serializer benchmarks, this drop was no longer observable before /
> >>> after the changes in mid February.
> >>> IMO, this is not critical as it does not affect the per-record
> >> performance
> >>> / throughput, and therefore should not block this release.
> >>>
> >>> On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <
> [hidden email]>
> >>> wrote:
> >>>
> >>>> By now, I'm reasonably sure that the test instabilities on the
> >> end-to-end
> >>>> test are only instabilities. I pushed changes to increase timeouts to
> >> make
> >>>> the tests more stable. As in any project, there will always be bugs
> but
> >> I
> >>>> think we could release this RC4 and be reasonably sure that it works
> >> well.
> >>>>
> >>>> Now, we only need to have the required number of PMC votes.
> >>>>
> >>>> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> >>>>> +1 （non-binding)
> >>>>>
> >>>>> • checked signature and checksum ok
> >>>>> • mvn clean package -DskipTests ok
> >>>>> • Run job on yarn ok
> >>>>> • Test state migration with POJO type (both heap and rocksdb) ok
> >>>>> • - 1.6 -> 1.8
> >>>>> • - 1.7 -> 1.8
> >>>>> • - 1.8 -> 1.8
> >>>>>
> >>>>>
> >>>>> Best, Congxian
> >>>>> On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>,
> >> wrote:
> >>>>>> +1 (non-binding)
> >>>>>>
> >>>>>> - checked JIRA release note
> >>>>>> - ran "mvn package -DskipTests"
> >>>>>> - checked signature and checksum
> >>>>>> - started a cluster locally and ran some examples in binary
> >>>>>> - checked web site announcement's PR
> >>>>>>
> >>>>>> Best,
> >>>>>> Vino
> >>>>>>
> >>>>>>
> >>>>>> Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
> >>>>>>
> >>>>>>> +1 (non-binding)
> >>>>>>>
> >>>>>>> - checked checksums and GPG files
> >>>>>>> - build from source successfully- run end-to-end precommit tests
> >>>>>>> successfully- run end-to-end nightly tests successfully
> >>>>>>> Xiaowei
> >>>>>>> On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> >>>> [hidden email]>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> +1 (non-binding)
> >>>>>>>
> >>>>>>> - Checked release notes: OK
> >>>>>>> - Checked sums and signatures: OK
> >>>>>>> - Source release
> >>>>>>> - contains no binaries: OK
> >>>>>>> - contains no 1.8-SNAPSHOT references: OK
> >>>>>>> - build from source: OK (8u101)
> >>>>>>> - mvn clean verify: OK (8u101)
> >>>>>>> - Binary release
> >>>>>>> - no examples appear to be missing
> >>>>>>> - started a cluster; WebUI reachable, example ran successfully
> >>>>>>> - end-to-end test (all but K8S and docker ones): OK (8u101)
> >>>>>>> - Repository appears to contain all expected artifacts
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Yu
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]>
> >> wrote:
> >>>>>>>
> >>>>>>>> +1 (non-binding)
> >>>>>>>>
> >>>>>>>> Checked items:
> >>>>>>>> - checked checksums and GPG files
> >>>>>>>> - verified that the source archives do not contains any binaries
> >>>>>>>> - checked that all POM files point to the same version
> >>>>>>>> - build from source successfully
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Kurt
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
> >>>> [hidden email]>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> +1 (non-binding)
> >>>>>>>>>
> >>>>>>>>> I tested RC4 with the following items:
> >>>>>>>>> - Maven Central Repository contains all artifacts
> >>>>>>>>> - Built the source with Maven (ensured all source files have
> >>>> Apache
> >>>>>>>>> headers), and executed built-in tests via "mvn clean verify"
> >>>>>>>>> - Manually executed the tests in IntelliJ IDE
> >>>>>>>>> - Verify that the quickstarts for Scala and Java are working
> >>>> with the
> >>>>>>>>> staging repository in IntelliJ
> >>>>>>>>> - Checked the benchmark results. The perf regression of
> >>>>>>>>> tuple-key-by/statebackend/tumblingWindow are gone, but the
> >>>> regression
> >>>>>>> on
> >>>>>>>>> serializer still exists.
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Shaoxuan
> >>>>>>>>>
> >>>>>>>>> On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
> >>>> [hidden email]
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Aljoscha, I think you are right, increase the timeout
> >>>> config will
> >>>>>>>> fix
> >>>>>>>>>> this issue. this depends on the resource of Travis. I would
> >>>> like
> >>>>>>> share
> >>>>>>>>>> some phenomenon during my test (not the flink problem) as
> >>>> follows:
> >>>>>>> :-)
> >>>>>>>>>>
> >>>>>>>>>> During my testing, `mvn clean verify` and `nightly
> >> end-to-end
> >>>> test `
> >>>>>>>> both
> >>>>>>>>>> consume a lot of machine resources (especially
> >>>> memory/network), and
> >>>>>>> the
> >>>>>>>>>> network bandwidth requirements of `nightly end-to-end test `
> >>>> are also
> >>>>>>>>> very
> >>>>>>>>>> high. In China, need to use VPN acceleration (100~200Kb
> >> before
> >>>>>>>>>> acceleration, 3~4Mb after acceleration), I have encountered:
> >>>> [Avro
> >>>>>>>>>> Confluent Schema Registry nightly end-to-end test' failed
> >>>> after 18
> >>>>>>>>> minutes
> >>>>>>>>>> and 15 seconds! Test exited with exit Code 1] takes more
> >> than
> >>>> 18
> >>>>>>>> minutes,
> >>>>>>>>>> the download failed because the network bandwidth is not
> >>>> enough. and
> >>>>>>> it
> >>>>>>>>>> runs smoothly when using VPN acceleration. The overall
> >>>> end-to-end run
> >>>>>>>> was
> >>>>>>>>>> passed twice. The Docker resource configuration (CUPs 7,
> >> Mem:
> >>>> 28.7G,
> >>>>>>>>> Swap:
> >>>>>>>>>> 3.5G). See detail log here
> >>>>>>>>>> <
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> >>>>>>>>>>>
> >>>>>>>>>> .
> >>>>>>>>>>
> >>>>>>>>>> Just now, I had checked the Travis for your last commit
> >>>> (Increase
> >>>>>>>> startup
> >>>>>>>>>> timeout in end-to-end tests), in addition to the Cleanup
> >>>> phase, other
> >>>>>>>>>> phases are successful. here
> >>>>>>>>>> <https://travis-ci.org/apache/flink/builds/511071777>
> >>>>>>>>>>
> >>>>>>>>>> In order to verify that our speculation is accurate, I can
> >>>> help with
> >>>>>>> 10
> >>>>>>>>> and
> >>>>>>>>>> 20 seconds timeout config on my repo verification to see if
> >>>> 100%
> >>>>>>>>> recurring
> >>>>>>>>>> timeout problem. It is already running, we are waiting for
> >> the
> >>>>>>> result.
> >>>>>>>>>> 10seconds <
> >>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235749
> >>>>>>>>>
> >>>>>>>>>> 20seconds <
> >>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235598
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Jincheng
> >>>>>>>>>>
> >>>>>>>>>> Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
> >>>> 上午1:04写道：
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for the testing done so far!
> >>>>>>>>>>>
> >>>>>>>>>>> There has been quite some flakiness on Travis lately, see
> >>>> here:
> >>>>>>>>>>> https://travis-ci.org/apache/flink/branches <
> >>>>>>>>>>> https://travis-ci.org/apache/flink/branches>. I’m a bit
> >>>> hesitant
> >>>>>>> to
> >>>>>>>>>>> release in this state. Looking at the tests you can see
> >>>> that all of
> >>>>>>>> the
> >>>>>>>>>>> end-to-end tests fail because waiting for the dispatcher
> >> to
> >>>> come up
> >>>>>>>>> times
> >>>>>>>>>>> out. I also noticed that this usually takes about 5-8
> >>>> seconds on
> >>>>>>>>> Travis,
> >>>>>>>>>> so
> >>>>>>>>>>> a 10 second timeout might be a bit low. I pushed commits
> >> to
> >>>>>>> increase
> >>>>>>>>> that
> >>>>>>>>>>> to 20 secs. Let’s see what will happen.
> >>>>>>>>>>>
> >>>>>>>>>>> I’ll keep you posted!
> >>>>>>>>>>> Aljoscha
> >>>>>>>>>>>
> >>>>>>>>>>>> On 25. Mar 2019, at 13:13, jincheng sun <
> >>>>>>> [hidden email]>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Great thanks for preparing the RC4 of Flink 1.8.0,
> >>>> Aljoscha!
> >>>>>>>>>>>>
> >>>>>>>>>>>> +1 (non-binding)
> >>>>>>>>>>>>
> >>>>>>>>>>>> I checked the functional things as follows(Without
> >>>> performance
> >>>>>>>>>>>> verification):
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Checking Artifacts：
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1). Download the release source code - SUCCESS
> >>>>>>>>>>>> 2). Check Source release flink-1.8.0-src.tgz.sha512 -
> >>>> SUCCESS
> >>>>>>>>>>>> 3). Download the released JAR - SUCCESS
> >>>>>>>>>>>> 4). Check if checksums and GPG files match the
> >>>> corresponding
> >>>>>>>>> release
> >>>>>>>>>>>> files - SUCCESS.
> >>>>>>>>>>>> 5). Verify that the source archives do not contain any
> >>>>>>> binaries
> >>>>>>>> -
> >>>>>>>>>>>> SUCCESS.
> >>>>>>>>>>>> 6). Build the source with `mvn clean verify -DskipTests`
> >>>> to
> >>>>>>>> ensure
> >>>>>>>>>> all
> >>>>>>>>>>>> source files have Apache headers - SUCCESS
> >>>>>>>>>>>> 7). Check that all POM files point to the same version -
> >>>>>>> SUCCESS
> >>>>>>>>>>>> 8). Read the `README.md` file to ensure there is nothing
> >>>>>>>>> unexpected
> >>>>>>>>>> -
> >>>>>>>>>>>> SUCCESS
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Testing Larger Setups
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cluster Environment：7 nodes, jm 1024m, tm 4096m
> >>>>>>>>>>>> Testing Jobs: WordCount(Batch&Streaming),
> >>>>>>>>>> DataStreamAllroundTestProgram
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1). Use local&hdfs file systems for checkpoints -
> >> SUCCESS
> >>>>>>>>>>>> 2). Use hdfs file systems for input/output -SUCCESS
> >>>>>>>>>>>> 3). Run examples on YARN(with or without session) -
> >>>> SUCCESS
> >>>>>>>>>>>> 4). Test failover and recovery. - SUCCESS
> >>>>>>>>>>>> 5). Test incremental&non-incremental checkpoint -
> >> SUCCESS
> >>>>>>>>>>>> 6). Test connector - kafka -SUCCESS
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3. Testing Functionality
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1). Built-in tests(linux&mac os)
> >>>>>>>>>>>> - `mvn cealn verify` (some test timeout error and test
> >>>> case
> >>>>>>>> bug
> >>>>>>>>>> see
> >>>>>>>>>>>> FLINK-12001 <
> >>>> https://issues.apache.org/jira/browse/FLINK-12001>,
> >>>>>>>> all
> >>>>>>>>>> of
> >>>>>>>>>>>> them are not the blocker)
> >>>>>>>>>>>> - build for scala 2.11（mvn clean install -P scala-2.11
> >>>>>>>>>> -DskipTests）
> >>>>>>>>>>>> - SUCCESS
> >>>>>>>>>>>> - Run the scripted nightly end-to-end test - SUCCESS
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2). Quickstarts
> >>>>>>>>>>>> - Verify that the quickstarts for Scala with the staging
> >>>>>>>>>> repository
> >>>>>>>>>>>> in IntelliJ - SUCCESS
> >>>>>>>>>>>> - Verify that the quickstarts for Java with the staging
> >>>>>>>>> repository
> >>>>>>>>>>> in
> >>>>>>>>>>>> IntelliJ - SUCCESS
> >>>>>>>>>>>>
> >>>>>>>>>>>> 3). Simple Starter Experience and Use Cases
> >>>>>>>>>>>>
> >>>>>>>>>>>> - run all examples from IntelliJ IDE - SUCCESS
> >>>>>>>>>>>> - Start a local cluster and verify that the processes -
> >>>>>>>> SUCCESS
> >>>>>>>>>>>> a. Examine the *.out files (should be empty) and the log
> >>>>>>>> files
> >>>>>>>>>>>> (should contain no exceptions)
> >>>>>>>>>>>> b. Test for Linux, MacOS
> >>>>>>>>>>>> c. Shutdown and verify there are no exceptions in the
> >> log
> >>>>>>>>> output
> >>>>>>>>>>>> (after shutdown)
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Verify that the examples are running from both
> >>>> ./bin/flink
> >>>>>>>> and
> >>>>>>>>>>> from
> >>>>>>>>>>>> the web-based job submission tool(following items) -
> >>>> SUCCESS
> >>>>>>>>>>>> a. Start multiple task managers in the local cluster
> >>>>>>>>>>>> b. Change the flink-conf.yml to define more than one
> >> task
> >>>>>>>> slot
> >>>>>>>>>> (2)
> >>>>>>>>>>>> c. Run the examples with a parallelism > 1
> >>>>>>>>>>>> d. Examine the log output - no error messages should be
> >>>>>>>>>>> encountered
> >>>>>>>>>>>>
> >>>>>>>>>>>> 4. Review the PR
> >>>>>>>>>>>> - [Add 1.8 Release Blog Post] - Just a reminder, updated
> >>>> the
> >>>>>>>>>> release
> >>>>>>>>>>>> date to correct date before merging.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Jincheng
> >>>>>>>>>>>>
> >>>>>>>>>>>> Piotr Nowojski <[hidden email]> 于2019年3月25日周一
> >>>> 下午4:11写道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> +1 from my side. Previously spotted performance
> >>>> regression seems
> >>>>>>>> to
> >>>>>>>>> be
> >>>>>>>>>>>>> gone, or mostly gone.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Piotrek
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> >>>>>>> [hidden email]>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi everyone,
> >>>>>>>>>>>>>> Please review and vote on the release candidate 4
> >> for
> >>>> Flink
> >>>>>>>> 1.8.0,
> >>>>>>>>> as
> >>>>>>>>>>>>> follows:
> >>>>>>>>>>>>>> [ ] +1, Approve the release
> >>>>>>>>>>>>>> [ ] -1, Do not approve the release (please provide
> >>>> specific
> >>>>>>>>> comments)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The complete staging area is available for your
> >>>> review, which
> >>>>>>>>>> includes:
> >>>>>>>>>>>>>> * JIRA release notes [1],
> >>>>>>>>>>>>>> * the official Apache source release and binary
> >>>> convenience
> >>>>>>>>> releases
> >>>>>>>>>> to
> >>>>>>>>>>>>> be deployed to dist.apache.org [2], which are signed
> >>>> with the
> >>>>>>> key
> >>>>>>>>>> with
> >>>>>>>>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
> >>>> [3],
> >>>>>>>>>>>>>> * all artifacts to be deployed to the Maven Central
> >>>> Repository
> >>>>>>>> [4],
> >>>>>>>>>>>>>> * source code tag "release-1.8.0-rc4" [5],
> >>>>>>>>>>>>>> * website pull request listing the new release [6]
> >>>>>>>>>>>>>> * website pull request adding announcement blog post
> >>>> [7].
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The vote will be open for at least 72 hours. It is
> >>>> adopted by
> >>>>>>>>>> majority
> >>>>>>>>>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Aljoscha
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1]
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> >>>>>>>>>>>>>> [2]
> >>>>>>>> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> >>>>>>>>>>>>>> [3]
> >>>> https://dist.apache.org/repos/dist/release/flink/KEYS
> >>>>>>>>>>>>>> [4]
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapacheflink-1215
> >>>>>>>>>>>>>> [5]
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> >>>>>>>>>>>>>> [6] https://github.com/apache/flink-web/pull/180 <
> >>>>>>>>>>>>> https://github.com/apache/flink-web/pull/180>
> >>>>>>>>>>>>>> [7] https://github.com/apache/flink-web/pull/179 <
> >>>>>>>>>>>>> https://github.com/apache/flink-web/pull/179>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> P.S. The difference to the previous RCs is small,
> >> you
> >>>> can fetch
> >>>>>>>> the
> >>>>>>>>>>> tags
> >>>>>>>>>>>>> and do a "git log
> >> release-1.8.0-rc1..release-1.8.0-rc4”
> >>>> to see
> >>>>>>> the
> >>>>>>>>>>>>> difference in commits. Its fixes for the issues that
> >>>> led to the
> >>>>>>>>>>>>> cancellation of the previous RCs plus smaller fixes.
> >>>> Most
> >>>>>>>>>>>>> verification/testing that was carried out should apply
> >>>> as is to
> >>>>>>>> this
> >>>>>>>>>> RC.
> >>>>>>>>>>>>> Any functional verification that you did on previous
> >>>> RCs should
> >>>>>>>>>>> therefore
> >>>>>>>>>>>>> easily carry over to this one.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Richard Deurwaarder

Re: [VOTE] Release 1.8.0, release candidate #4

Hello @Aljoscha and @Rong,

I've described the problem in the mailing list[1] and on stackoverflow[2]
before. But the gist is: If there's a firewall between the yarn cluster and
the machine submitting the job, we need to be able to set a fixed port (or
range of ports) for REST communication with the jobmanager.

It is a regression in the sense that on 1.5 (and 1.6 I believe?) it was
possible to work around this by using the legacy mode (non flip-6), but on
1.7 and now 1.8 this is not possible.

I've created FLINK-12075 <https://issues.apache.org/jira/browse/FLINK-12075>
for it, I have not made it blocking yet as it is not strictly a regression
with regards to 1.7. Perhaps you guys can better determine if you want this
added in RC5.

Regards,

Richard

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Submitting-job-to-Flink-on-yarn-timesout-on-flip-6-1-5-x-td26199.html#a26383
[2] https://stackoverflow.com/q/54771637/988324

On Sat, Mar 30, 2019 at 7:24 PM Rong Rong <[hidden email]> wrote:

> Hi @Aljoscha,
>
> Based on the previous commit [1] that adds the random port selection code,
> it seems like the important part is to unset whatever 'rest.port' setting
> previously done. I don't think the current way of setting the BIND_PORT
> actually overrides any existing PORT setting. However, I wasn't able to
> find any test that is related, maybe @Till can provide more insight here?
>
> Maybe @Richard can provide more detail on the YARN run command used to
> reproduce the problem?
>
> Thanks,
> Rong
>
> [1]
>
> https://github.com/apache/flink/commit/dbe0e8286d76a5facdb49589b638b87dbde80178#diff-487838863ab693af7008f04cb3359be3R117
>
> On Sat, Mar 30, 2019 at 5:51 AM Aljoscha Krettek <[hidden email]>
> wrote:
>
> > @Richard Did this work for you previously? From the change, it seems that
> > the port was always set to 0 on YARN even before.
> >
> > > On 28. Mar 2019, at 16:13, Richard Deurwaarder <[hidden email]>
> wrote:
> > >
> > > -1 (non-binding)
> > >
> > > - Ran integration tests locally (1000+) of our flink job, all
> succeeded.
> > > - Attempted to run job on hadoop, failed. It failed because we have a
> > > firewall in place and we cannot set the rest port to a specific
> port/port
> > > range.
> > > Unless I am mistaken, it seems like FLINK-11081 broke the possibility
> of
> > > setting a REST port when running on yarn (
> > >
> >
> https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
> > > )
> > > Code-wise it seems rather straightforward to fix but I am unsure about
> > the
> > > reason why this is hard-coded to 0 and what the impact would be.
> > >
> > > It would benefit us greatly if a fix for this could make it to 1.8.0.
> > >
> > > Regards,
> > >
> > > Richard
> > >
> > > On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai <
> [hidden email]
> > >
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Functional checks:
> > >>
> > >> - Built Flink from source (`mvn clean verify`) locally, with success
> > >> - Ran end-to-end tests locally for 5 times in a loop, no attempts
> failed
> > >> (Hadoop 2.8.4, Scala 2.12)
> > >> - Manually tested state schema evolution for POJO. Besides the tests
> > that
> > >> @Congxian already did, additionally tested evolution cases with POJO
> > >> subclasses + non-registered POJOs.
> > >> - Manually tested migration of Scala stateful jobs that use case
> > classes /
> > >> Scala collections as state types, performing the migration across
> Scala
> > >> 2.11 to Scala 2.12.
> > >> - Reviewed release announcement PR
> > >>
> > >> Misc / legal checks:
> > >>
> > >> - checked checksums and signatures
> > >> - No binaries in source distribution
> > >> - Staging area does not seem to have any missing artifacts
> > >>
> > >> Cheers,
> > >> Gordon
> > >>
> > >> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai <
> > [hidden email]>
> > >> wrote:
> > >>
> > >>> @Shaoxuan
> > >>>
> > >>> The drop in the serializerAvro benchmark, as explained earlier in
> > >> previous
> > >>> voting threads of earlier RCs, was due to a slower job initialization
> > >> phase
> > >>> caused by slower deserialization of the AvroSerializer.
> > >>> Piotr also pointed out that after the number of records was increased
> > in
> > >>> the serializer benchmarks, this drop was no longer observable before
> /
> > >>> after the changes in mid February.
> > >>> IMO, this is not critical as it does not affect the per-record
> > >> performance
> > >>> / throughput, and therefore should not block this release.
> > >>>
> > >>> On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek <
> > [hidden email]>
> > >>> wrote:
> > >>>
> > >>>> By now, I'm reasonably sure that the test instabilities on the
> > >> end-to-end
> > >>>> test are only instabilities. I pushed changes to increase timeouts
> to
> > >> make
> > >>>> the tests more stable. As in any project, there will always be bugs
> > but
> > >> I
> > >>>> think we could release this RC4 and be reasonably sure that it works
> > >> well.
> > >>>>
> > >>>> Now, we only need to have the required number of PMC votes.
> > >>>>
> > >>>> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> > >>>>> +1 （non-binding)
> > >>>>>
> > >>>>> • checked signature and checksum ok
> > >>>>> • mvn clean package -DskipTests ok
> > >>>>> • Run job on yarn ok
> > >>>>> • Test state migration with POJO type (both heap and rocksdb) ok
> > >>>>> • - 1.6 -> 1.8
> > >>>>> • - 1.7 -> 1.8
> > >>>>> • - 1.8 -> 1.8
> > >>>>>
> > >>>>>
> > >>>>> Best, Congxian
> > >>>>> On Mar 27, 2019, 10:26 +0800, vino yang <[hidden email]>,
> > >> wrote:
> > >>>>>> +1 (non-binding)
> > >>>>>>
> > >>>>>> - checked JIRA release note
> > >>>>>> - ran "mvn package -DskipTests"
> > >>>>>> - checked signature and checksum
> > >>>>>> - started a cluster locally and ran some examples in binary
> > >>>>>> - checked web site announcement's PR
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Vino
> > >>>>>>
> > >>>>>>
> > >>>>>> Xiaowei Jiang <[hidden email]> 于2019年3月26日周二下午8:20写道：
> > >>>>>>
> > >>>>>>> +1 (non-binding)
> > >>>>>>>
> > >>>>>>> - checked checksums and GPG files
> > >>>>>>> - build from source successfully- run end-to-end precommit tests
> > >>>>>>> successfully- run end-to-end nightly tests successfully
> > >>>>>>> Xiaowei
> > >>>>>>> On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> > >>>> [hidden email]>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> +1 (non-binding)
> > >>>>>>>
> > >>>>>>> - Checked release notes: OK
> > >>>>>>> - Checked sums and signatures: OK
> > >>>>>>> - Source release
> > >>>>>>> - contains no binaries: OK
> > >>>>>>> - contains no 1.8-SNAPSHOT references: OK
> > >>>>>>> - build from source: OK (8u101)
> > >>>>>>> - mvn clean verify: OK (8u101)
> > >>>>>>> - Binary release
> > >>>>>>> - no examples appear to be missing
> > >>>>>>> - started a cluster; WebUI reachable, example ran successfully
> > >>>>>>> - end-to-end test (all but K8S and docker ones): OK (8u101)
> > >>>>>>> - Repository appears to contain all expected artifacts
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Yu
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, 26 Mar 2019 at 14:28, Kurt Young <[hidden email]>
> > >> wrote:
> > >>>>>>>
> > >>>>>>>> +1 (non-binding)
> > >>>>>>>>
> > >>>>>>>> Checked items:
> > >>>>>>>> - checked checksums and GPG files
> > >>>>>>>> - verified that the source archives do not contains any binaries
> > >>>>>>>> - checked that all POM files point to the same version
> > >>>>>>>> - build from source successfully
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Kurt
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
> > >>>> [hidden email]>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> +1 (non-binding)
> > >>>>>>>>>
> > >>>>>>>>> I tested RC4 with the following items:
> > >>>>>>>>> - Maven Central Repository contains all artifacts
> > >>>>>>>>> - Built the source with Maven (ensured all source files have
> > >>>> Apache
> > >>>>>>>>> headers), and executed built-in tests via "mvn clean verify"
> > >>>>>>>>> - Manually executed the tests in IntelliJ IDE
> > >>>>>>>>> - Verify that the quickstarts for Scala and Java are working
> > >>>> with the
> > >>>>>>>>> staging repository in IntelliJ
> > >>>>>>>>> - Checked the benchmark results. The perf regression of
> > >>>>>>>>> tuple-key-by/statebackend/tumblingWindow are gone, but the
> > >>>> regression
> > >>>>>>> on
> > >>>>>>>>> serializer still exists.
> > >>>>>>>>>
> > >>>>>>>>> Regards,
> > >>>>>>>>> Shaoxuan
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
> > >>>> [hidden email]
> > >>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Aljoscha, I think you are right, increase the timeout
> > >>>> config will
> > >>>>>>>> fix
> > >>>>>>>>>> this issue. this depends on the resource of Travis. I would
> > >>>> like
> > >>>>>>> share
> > >>>>>>>>>> some phenomenon during my test (not the flink problem) as
> > >>>> follows:
> > >>>>>>> :-)
> > >>>>>>>>>>
> > >>>>>>>>>> During my testing, `mvn clean verify` and `nightly
> > >> end-to-end
> > >>>> test `
> > >>>>>>>> both
> > >>>>>>>>>> consume a lot of machine resources (especially
> > >>>> memory/network), and
> > >>>>>>> the
> > >>>>>>>>>> network bandwidth requirements of `nightly end-to-end test `
> > >>>> are also
> > >>>>>>>>> very
> > >>>>>>>>>> high. In China, need to use VPN acceleration (100~200Kb
> > >> before
> > >>>>>>>>>> acceleration, 3~4Mb after acceleration), I have encountered:
> > >>>> [Avro
> > >>>>>>>>>> Confluent Schema Registry nightly end-to-end test' failed
> > >>>> after 18
> > >>>>>>>>> minutes
> > >>>>>>>>>> and 15 seconds! Test exited with exit Code 1] takes more
> > >> than
> > >>>> 18
> > >>>>>>>> minutes,
> > >>>>>>>>>> the download failed because the network bandwidth is not
> > >>>> enough. and
> > >>>>>>> it
> > >>>>>>>>>> runs smoothly when using VPN acceleration. The overall
> > >>>> end-to-end run
> > >>>>>>>> was
> > >>>>>>>>>> passed twice. The Docker resource configuration (CUPs 7,
> > >> Mem:
> > >>>> 28.7G,
> > >>>>>>>>> Swap:
> > >>>>>>>>>> 3.5G). See detail log here
> > >>>>>>>>>> <
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://docs.google.com/document/d/1CcyTCyZmMmP57pkKv4drjSuxW61_u78HR3q1fJJODMw/edit?usp=sharing
> > >>>>>>>>>>>
> > >>>>>>>>>> .
> > >>>>>>>>>>
> > >>>>>>>>>> Just now, I had checked the Travis for your last commit
> > >>>> (Increase
> > >>>>>>>> startup
> > >>>>>>>>>> timeout in end-to-end tests), in addition to the Cleanup
> > >>>> phase, other
> > >>>>>>>>>> phases are successful. here
> > >>>>>>>>>> <https://travis-ci.org/apache/flink/builds/511071777>
> > >>>>>>>>>>
> > >>>>>>>>>> In order to verify that our speculation is accurate, I can
> > >>>> help with
> > >>>>>>> 10
> > >>>>>>>>> and
> > >>>>>>>>>> 20 seconds timeout config on my repo verification to see if
> > >>>> 100%
> > >>>>>>>>> recurring
> > >>>>>>>>>> timeout problem. It is already running, we are waiting for
> > >> the
> > >>>>>>> result.
> > >>>>>>>>>> 10seconds <
> > >>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235749
> > >>>>>>>>>
> > >>>>>>>>>> 20seconds <
> > >>>>>>> https://travis-ci.org/sunjincheng121/flink/builds/511235598
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Jincheng
> > >>>>>>>>>>
> > >>>>>>>>>> Aljoscha Krettek <[hidden email]> 于2019年3月26日周二
> > >>>> 上午1:04写道：
> > >>>>>>>>>>
> > >>>>>>>>>>> Thanks for the testing done so far!
> > >>>>>>>>>>>
> > >>>>>>>>>>> There has been quite some flakiness on Travis lately, see
> > >>>> here:
> > >>>>>>>>>>> https://travis-ci.org/apache/flink/branches <
> > >>>>>>>>>>> https://travis-ci.org/apache/flink/branches>. I’m a bit
> > >>>> hesitant
> > >>>>>>> to
> > >>>>>>>>>>> release in this state. Looking at the tests you can see
> > >>>> that all of
> > >>>>>>>> the
> > >>>>>>>>>>> end-to-end tests fail because waiting for the dispatcher
> > >> to
> > >>>> come up
> > >>>>>>>>> times
> > >>>>>>>>>>> out. I also noticed that this usually takes about 5-8
> > >>>> seconds on
> > >>>>>>>>> Travis,
> > >>>>>>>>>> so
> > >>>>>>>>>>> a 10 second timeout might be a bit low. I pushed commits
> > >> to
> > >>>>>>> increase
> > >>>>>>>>> that
> > >>>>>>>>>>> to 20 secs. Let’s see what will happen.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I’ll keep you posted!
> > >>>>>>>>>>> Aljoscha
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On 25. Mar 2019, at 13:13, jincheng sun <
> > >>>>>>> [hidden email]>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Great thanks for preparing the RC4 of Flink 1.8.0,
> > >>>> Aljoscha!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> +1 (non-binding)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I checked the functional things as follows(Without
> > >>>> performance
> > >>>>>>>>>>>> verification):
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Checking Artifacts：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1). Download the release source code - SUCCESS
> > >>>>>>>>>>>> 2). Check Source release flink-1.8.0-src.tgz.sha512 -
> > >>>> SUCCESS
> > >>>>>>>>>>>> 3). Download the released JAR - SUCCESS
> > >>>>>>>>>>>> 4). Check if checksums and GPG files match the
> > >>>> corresponding
> > >>>>>>>>> release
> > >>>>>>>>>>>> files - SUCCESS.
> > >>>>>>>>>>>> 5). Verify that the source archives do not contain any
> > >>>>>>> binaries
> > >>>>>>>> -
> > >>>>>>>>>>>> SUCCESS.
> > >>>>>>>>>>>> 6). Build the source with `mvn clean verify -DskipTests`
> > >>>> to
> > >>>>>>>> ensure
> > >>>>>>>>>> all
> > >>>>>>>>>>>> source files have Apache headers - SUCCESS
> > >>>>>>>>>>>> 7). Check that all POM files point to the same version -
> > >>>>>>> SUCCESS
> > >>>>>>>>>>>> 8). Read the `README.md` file to ensure there is nothing
> > >>>>>>>>> unexpected
> > >>>>>>>>>> -
> > >>>>>>>>>>>> SUCCESS
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. Testing Larger Setups
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cluster Environment：7 nodes, jm 1024m, tm 4096m
> > >>>>>>>>>>>> Testing Jobs: WordCount(Batch&Streaming),
> > >>>>>>>>>> DataStreamAllroundTestProgram
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1). Use local&hdfs file systems for checkpoints -
> > >> SUCCESS
> > >>>>>>>>>>>> 2). Use hdfs file systems for input/output -SUCCESS
> > >>>>>>>>>>>> 3). Run examples on YARN(with or without session) -
> > >>>> SUCCESS
> > >>>>>>>>>>>> 4). Test failover and recovery. - SUCCESS
> > >>>>>>>>>>>> 5). Test incremental&non-incremental checkpoint -
> > >> SUCCESS
> > >>>>>>>>>>>> 6). Test connector - kafka -SUCCESS
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 3. Testing Functionality
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1). Built-in tests(linux&mac os)
> > >>>>>>>>>>>> - `mvn cealn verify` (some test timeout error and test
> > >>>> case
> > >>>>>>>> bug
> > >>>>>>>>>> see
> > >>>>>>>>>>>> FLINK-12001 <
> > >>>> https://issues.apache.org/jira/browse/FLINK-12001>,
> > >>>>>>>> all
> > >>>>>>>>>> of
> > >>>>>>>>>>>> them are not the blocker)
> > >>>>>>>>>>>> - build for scala 2.11（mvn clean install -P scala-2.11
> > >>>>>>>>>> -DskipTests）
> > >>>>>>>>>>>> - SUCCESS
> > >>>>>>>>>>>> - Run the scripted nightly end-to-end test - SUCCESS
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2). Quickstarts
> > >>>>>>>>>>>> - Verify that the quickstarts for Scala with the staging
> > >>>>>>>>>> repository
> > >>>>>>>>>>>> in IntelliJ - SUCCESS
> > >>>>>>>>>>>> - Verify that the quickstarts for Java with the staging
> > >>>>>>>>> repository
> > >>>>>>>>>>> in
> > >>>>>>>>>>>> IntelliJ - SUCCESS
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 3). Simple Starter Experience and Use Cases
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - run all examples from IntelliJ IDE - SUCCESS
> > >>>>>>>>>>>> - Start a local cluster and verify that the processes -
> > >>>>>>>> SUCCESS
> > >>>>>>>>>>>> a. Examine the *.out files (should be empty) and the log
> > >>>>>>>> files
> > >>>>>>>>>>>> (should contain no exceptions)
> > >>>>>>>>>>>> b. Test for Linux, MacOS
> > >>>>>>>>>>>> c. Shutdown and verify there are no exceptions in the
> > >> log
> > >>>>>>>>> output
> > >>>>>>>>>>>> (after shutdown)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Verify that the examples are running from both
> > >>>> ./bin/flink
> > >>>>>>>> and
> > >>>>>>>>>>> from
> > >>>>>>>>>>>> the web-based job submission tool(following items) -
> > >>>> SUCCESS
> > >>>>>>>>>>>> a. Start multiple task managers in the local cluster
> > >>>>>>>>>>>> b. Change the flink-conf.yml to define more than one
> > >> task
> > >>>>>>>> slot
> > >>>>>>>>>> (2)
> > >>>>>>>>>>>> c. Run the examples with a parallelism > 1
> > >>>>>>>>>>>> d. Examine the log output - no error messages should be
> > >>>>>>>>>>> encountered
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 4. Review the PR
> > >>>>>>>>>>>> - [Add 1.8 Release Blog Post] - Just a reminder, updated
> > >>>> the
> > >>>>>>>>>> release
> > >>>>>>>>>>>> date to correct date before merging.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>> Jincheng
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Piotr Nowojski <[hidden email]> 于2019年3月25日周一
> > >>>> 下午4:11写道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> +1 from my side. Previously spotted performance
> > >>>> regression seems
> > >>>>>>>> to
> > >>>>>>>>> be
> > >>>>>>>>>>>>> gone, or mostly gone.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Piotrek
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 21 Mar 2019, at 17:52, Aljoscha Krettek <
> > >>>>>>> [hidden email]>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi everyone,
> > >>>>>>>>>>>>>> Please review and vote on the release candidate 4
> > >> for
> > >>>> Flink
> > >>>>>>>> 1.8.0,
> > >>>>>>>>> as
> > >>>>>>>>>>>>> follows:
> > >>>>>>>>>>>>>> [ ] +1, Approve the release
> > >>>>>>>>>>>>>> [ ] -1, Do not approve the release (please provide
> > >>>> specific
> > >>>>>>>>> comments)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The complete staging area is available for your
> > >>>> review, which
> > >>>>>>>>>> includes:
> > >>>>>>>>>>>>>> * JIRA release notes [1],
> > >>>>>>>>>>>>>> * the official Apache source release and binary
> > >>>> convenience
> > >>>>>>>>> releases
> > >>>>>>>>>> to
> > >>>>>>>>>>>>> be deployed to dist.apache.org [2], which are signed
> > >>>> with the
> > >>>>>>> key
> > >>>>>>>>>> with
> > >>>>>>>>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293
> > >>>> [3],
> > >>>>>>>>>>>>>> * all artifacts to be deployed to the Maven Central
> > >>>> Repository
> > >>>>>>>> [4],
> > >>>>>>>>>>>>>> * source code tag "release-1.8.0-rc4" [5],
> > >>>>>>>>>>>>>> * website pull request listing the new release [6]
> > >>>>>>>>>>>>>> * website pull request adding announcement blog post
> > >>>> [7].
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The vote will be open for at least 72 hours. It is
> > >>>> adopted by
> > >>>>>>>>>> majority
> > >>>>>>>>>>>>> approval, with at least 3 PMC affirmative votes.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>> Aljoscha
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [1]
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274
> > >>>>>>>>>>>>>> [2]
> > >>>>>>>> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc4/
> > >>>>>>>>>>>>>> [3]
> > >>>> https://dist.apache.org/repos/dist/release/flink/KEYS
> > >>>>>>>>>>>>>> [4]
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>
> > https://repository.apache.org/content/repositories/orgapacheflink-1215
> > >>>>>>>>>>>>>> [5]
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>
> > >>
> >
> https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=c650befc10c8bb6cc4b007ae250b7b2173046145
> > >>>>>>>>>>>>>> [6] https://github.com/apache/flink-web/pull/180 <
> > >>>>>>>>>>>>> https://github.com/apache/flink-web/pull/180>
> > >>>>>>>>>>>>>> [7] https://github.com/apache/flink-web/pull/179 <
> > >>>>>>>>>>>>> https://github.com/apache/flink-web/pull/179>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> P.S. The difference to the previous RCs is small,
> > >> you
> > >>>> can fetch
> > >>>>>>>> the
> > >>>>>>>>>>> tags
> > >>>>>>>>>>>>> and do a "git log
> > >> release-1.8.0-rc1..release-1.8.0-rc4”
> > >>>> to see
> > >>>>>>> the
> > >>>>>>>>>>>>> difference in commits. Its fixes for the issues that
> > >>>> led to the
> > >>>>>>>>>>>>> cancellation of the previous RCs plus smaller fixes.
> > >>>> Most
> > >>>>>>>>>>>>> verification/testing that was carried out should apply
> > >>>> as is to
> > >>>>>>>> this
> > >>>>>>>>>> RC.
> > >>>>>>>>>>>>> Any functional verification that you did on previous
> > >>>> RCs should
> > >>>>>>>>>>> therefore
> > >>>>>>>>>>>>> easily carry over to this one.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>