(DEPRECATED) Apache Flink Mailing List archive.

First development steps, broken build workarounds?

Classic

List

Threaded

6 messages Options

bytesandwich

First development steps, broken build workarounds?

Hi All,

How do you start a fresh ticket in the flink repo for you guys' workflows? Master doesn't seem to build `mvn clean package` after trying a few different commits and a few tweaks to fork settings.

Is there maybe some workaround part of the development setup? Maybe it's just windows linux subsystem that's the issue?

Specifically, I've tried building a few commits on master as a first development project and I keep seeing a huge wall of: "The forked VM terminated without properly saying goodbye".

It also looks like the build has been failing for a few months? Am I seeing that right? http://mail-archives.apache.org/mod_mbox/flink-builds/202101.mbox/browser That's why I think maybe we have a regular practices of skipping some tests or setting some flags or something like that?

It looks like my builds are repeatedly dying in the "Runtime" test suite.

# Specific Failure:
```
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] BlobServerRangeTest.testOnePortAvailable:106
Expected: (a value equal to or greater than <50000> and a value less than or equal to <50050>)
but: a value equal to or greater than <50000> <49877> was less than <50000>
[ERROR] BlobServerRangeTest.testPortUnavailable Expected exception: java.io.IOException
[ERROR] BootstrapToolsTest.testActorSystemInstantiationFailureWhenPortOccupied:528 Expected to fail with a BindException
[ERROR] RestClientTest.testConnectionTimeout:78
Expected: an instance of org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
but: <org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /240.0.0.0:80> is a org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException
[ERROR] TaskManagerRunnerStartupTest.testStartupWhenNetworkStackFailsToInitialize:163 Should throw IOException when the network stack cannot be initialized.
[INFO]
[ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
```

# Stopping at suite: Runtime
```
[ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Flink : 1.13-SNAPSHOT:
[INFO]
[INFO] Flink : Tools : Force Shading ...................... SUCCESS [ 11.719 s]
[INFO] Flink : ............................................ SUCCESS [ 22.373 s]
[INFO] Flink : Annotations ................................ SUCCESS [ 4.312 s]
[INFO] Flink : Test utils : ............................... SUCCESS [ 0.384 s]
[INFO] Flink : Test utils : Junit ......................... SUCCESS [ 6.516 s]
[INFO] Flink : Metrics : .................................. SUCCESS [ 0.461 s]
[INFO] Flink : Metrics : Core ............................. SUCCESS [ 4.643 s]
[INFO] Flink : Core ....................................... SUCCESS [02:00 min]
[INFO] Flink : Java ....................................... SUCCESS [ 48.244 s]
[INFO] Flink : Queryable state : .......................... SUCCESS [ 0.315 s]
[INFO] Flink : Queryable state : Client Java .............. SUCCESS [ 5.269 s]
[INFO] Flink : FileSystems : .............................. SUCCESS [ 0.461 s]
[INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 17.152 s]

[INFO] Flink : Runtime .................................... FAILURE [09:08 min]

[INFO] Flink : Scala ...................................... SKIPPED
[INFO] Flink : FileSystems : Mapr FS ...................... SKIPPED
[INFO] Flink : FileSystems : Hadoop FS shaded ............. SKIPPED
[INFO] Flink : FileSystems : S3 FS Base ................... SKIPPED
--
```

# Many many crashed vms
```
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /mnt/c/Users/jack/src/flink/flink-runtime && /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms256m -Xmx2048m -Dmvn.forkNumber=2 -XX:+UseG1GC -jar /mnt/c/Users/jack/src/flink/flink-runtime/target/surefire/surefirebooter15675874354821454443.jar /mnt/c/Users/jack/src/flink/flink-runtime/target/surefire 2021-02-05T17-53-07_512-jvmRun2 surefire14636922785903152248tmp surefire_7126512113205932057927tmp
[ERROR] Process Exit Code: 0
```

Jack Phelan
[hidden email]

郁蓝

回复：First development steps, broken build workarounds?

Hi Jack,

I've just packaged a brand new dist from source several days ago.

I think you should do `mvn clean install -DskipTest` from the project root and see the output.

In fact the project **NEEDS** some workaround to compile, for instance it will complain about `@ant-design/icons-angular-icons` is missing after `npm install` is executed, you have to run `ng add @ant-design/icons-angular` after that and every thing goes well.

PS: If you have modified your `$HOME/.m2/settings.xml` to third-party mirror (e.g. aliyun in mainland China), proberbly it will fail because some dependencies are missing, you'd better put it to another place.

Wish would help

------------------ 原始邮件 ------------------
发件人: "dev" <[hidden email]>;
发送时间: 2021年2月6日(星期六) 中午11:36
收件人: "dev"<[hidden email]>;

主题: First development steps, broken build workarounds?

Hi All,

How do you start a fresh ticket in the flink repo for you guys' workflows? Master doesn't seem to build `mvn clean package` after trying a few different commits and a few tweaks to fork settings.

Is there maybe some workaround part of the development setup? Maybe it's just windows linux subsystem that's the issue?

Specifically, I've tried building a few commits on master as a first development project and I keep seeing a huge wall of: "The forked VM terminated without properly saying goodbye".

It also looks like the build has been failing for a few months? Am I seeing that right? http://mail-archives.apache.org/mod_mbox/flink-builds/202101.mbox/browser That's why I think maybe we have a regular practices of skipping some tests or setting some flags or something like that?

It looks like my builds are repeatedly dying in the "Runtime" test suite.

# Specific Failure:
```
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   BlobServerRangeTest.testOnePortAvailable:106
Expected: (a value equal to or greater than <50000> and a value less than or equal to <50050>)
     but: a value equal to or greater than <50000> <49877> was less than <50000>
[ERROR]   BlobServerRangeTest.testPortUnavailable Expected exception: java.io.IOException
[ERROR]   BootstrapToolsTest.testActorSystemInstantiationFailureWhenPortOccupied:528 Expected to fail with a BindException
[ERROR]   RestClientTest.testConnectionTimeout:78
Expected: an instance of org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
     but: <org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /240.0.0.0:80> is a org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException
[ERROR]   TaskManagerRunnerStartupTest.testStartupWhenNetworkStackFailsToInitialize:163 Should throw IOException when the network stack cannot be initialized.
[INFO]
[ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
```

# Stopping at suite: Runtime
```
[ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Flink : 1.13-SNAPSHOT:
[INFO]
[INFO] Flink : Tools : Force Shading ...................... SUCCESS [ 11.719 s]
[INFO] Flink : ............................................ SUCCESS [ 22.373 s]
[INFO] Flink : Annotations ................................ SUCCESS [  4.312 s]
[INFO] Flink : Test utils : ............................... SUCCESS [  0.384 s]
[INFO] Flink : Test utils : Junit ......................... SUCCESS [  6.516 s]
[INFO] Flink : Metrics : .................................. SUCCESS [  0.461 s]
[INFO] Flink : Metrics : Core ............................. SUCCESS [  4.643 s]
[INFO] Flink : Core ....................................... SUCCESS [02:00 min]
[INFO] Flink : Java ....................................... SUCCESS [ 48.244 s]
[INFO] Flink : Queryable state : .......................... SUCCESS [  0.315 s]
[INFO] Flink : Queryable state : Client Java .............. SUCCESS [  5.269 s]
[INFO] Flink : FileSystems : .............................. SUCCESS [  0.461 s]
[INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 17.152 s]

[INFO] Flink : Runtime .................................... FAILURE [09:08 min]

[INFO] Flink : Scala ...................................... SKIPPED
[INFO] Flink : FileSystems : Mapr FS ...................... SKIPPED
[INFO] Flink : FileSystems : Hadoop FS shaded ............. SKIPPED
[INFO] Flink : FileSystems : S3 FS Base ................... SKIPPED
--
```

# Many many crashed vms
```
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /mnt/c/Users/jack/src/flink/flink-runtime && /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Xms256m -Xmx2048m -Dmvn.forkNumber=2 -XX:+UseG1GC -jar /mnt/c/Users/jack/src/flink/flink-runtime/target/surefire/surefirebooter15675874354821454443.jar /mnt/c/Users/jack/src/flink/flink-runtime/target/surefire 2021-02-05T17-53-07_512-jvmRun2 surefire14636922785903152248tmp surefire_7126512113205932057927tmp
[ERROR] Process Exit Code: 0
```

  Jack Phelan
  [hidden email]

Chesnay Schepler-3

Re: First development steps, broken build workarounds?

In reply to this post by bytesandwich

When running the tests in WSL you will run into 2 issues:
a) building the WebUI can be unbearably so such that it pretty much
never finishes; you may want to activate the skip-webui-build profile
b) certain tests check how port conflicts are being handled, but WSL has
broken behavior in that regard. Multiple sockets can be created on the
same port without throwing an exception. As such, certain tests will
categorically fail on WSL. There is no switch to turn these tests off;
because bar myself I'm not aware of any other Flink developer using
Windows/WSL.

2/6/2021 4:36 AM, Jack Phelan wrote:

> Hi All,
>
> How do you start a fresh ticket in the flink repo for you guys' workflows? Master doesn't seem to build `mvn clean package` after trying a few different commits and a few tweaks to fork settings.
>
> Is there maybe some workaround part of the development setup? Maybe it's just windows linux subsystem that's the issue?
>
> Specifically, I've tried building a few commits on master as a first development project and I keep seeing a huge wall of: "The forked VM terminated without properly saying goodbye".
>
> It also looks like the build has been failing for a few months? Am I seeing that right? http://mail-archives.apache.org/mod_mbox/flink-builds/202101.mbox/browser That's why I think maybe we have a regular practices of skipping some tests or setting some flags or something like that?
>
> It looks like my builds are repeatedly dying in the "Runtime" test suite.
>
> # Specific Failure:
> ```
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR] BlobServerRangeTest.testOnePortAvailable:106
> Expected: (a value equal to or greater than <50000> and a value less than or equal to <50050>)
> but: a value equal to or greater than <50000> <49877> was less than <50000>
> [ERROR] BlobServerRangeTest.testPortUnavailable Expected exception: java.io.IOException
> [ERROR] BootstrapToolsTest.testActorSystemInstantiationFailureWhenPortOccupied:528 Expected to fail with a BindException
> [ERROR] RestClientTest.testConnectionTimeout:78
> Expected: an instance of org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
> but: <org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /240.0.0.0:80> is a org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException
> [ERROR] TaskManagerRunnerStartupTest.testStartupWhenNetworkStackFailsToInitialize:163 Should throw IOException when the network stack cannot be initialized.
> [INFO]
> [ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
> ```
>
> # Stopping at suite: Runtime
> ```
> [ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary for Flink : 1.13-SNAPSHOT:
> [INFO]
> [INFO] Flink : Tools : Force Shading ...................... SUCCESS [ 11.719 s]
> [INFO] Flink : ............................................ SUCCESS [ 22.373 s]
> [INFO] Flink : Annotations ................................ SUCCESS [ 4.312 s]
> [INFO] Flink : Test utils : ............................... SUCCESS [ 0.384 s]
> [INFO] Flink : Test utils : Junit ......................... SUCCESS [ 6.516 s]
> [INFO] Flink : Metrics : .................................. SUCCESS [ 0.461 s]
> [INFO] Flink : Metrics : Core ............................. SUCCESS [ 4.643 s]
> [INFO] Flink : Core ....................................... SUCCESS [02:00 min]
> [INFO] Flink : Java ....................................... SUCCESS [ 48.244 s]
> [INFO] Flink : Queryable state : .......................... SUCCESS [ 0.315 s]
> [INFO] Flink : Queryable state : Client Java .............. SUCCESS [ 5.269 s]
> [INFO] Flink : FileSystems : .............................. SUCCESS [ 0.461 s]
> [INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 17.152 s]
>
>
> [INFO] Flink : Runtime .................................... FAILURE [09:08 min]
>
>
> [INFO] Flink : Scala ...................................... SKIPPED
> [INFO] Flink : FileSystems : Mapr FS ...................... SKIPPED
> [INFO] Flink : FileSystems : Hadoop FS shaded ............. SKIPPED
> [INFO] Flink : FileSystems : S3 FS Base ................... SKIPPED

肯定会有

回复： First development steps, broken build workarounds?

Hi Guys,

I want to contribute to Apache Flink.
Would you please give me the permission as a contributor?
My JIRA ID is tony.xiao.

------------------ 原始邮件 ------------------
发件人: "dev" <[hidden email]>;
发送时间: 2021年2月6日(星期六) 晚上9:52
收件人: "dev"<[hidden email]>;"Jack Phelan"<[hidden email]>;

主题: Re: First development steps, broken build workarounds?

When running the tests in WSL you will run into 2 issues:
a) building the WebUI can be unbearably so such that it pretty much
never finishes; you may want to activate the skip-webui-build profile
b) certain tests check how port conflicts are being handled, but WSL has
broken behavior in that regard. Multiple sockets can be created on the
same port without throwing an exception. As such, certain tests will
categorically fail on WSL. There is no switch to turn these tests off;
because bar myself I'm not aware of any other Flink developer using
Windows/WSL.

2/6/2021 4:36 AM, Jack Phelan wrote:
> Hi All,
>
> How do you start a fresh ticket in the flink repo for you guys' workflows? Master doesn't seem to build `mvn clean package` after trying a few different commits and a few tweaks to fork settings.
>
> Is there maybe some workaround part of the development setup? Maybe it's just windows linux subsystem that's the issue?
>
> Specifically, I've tried building a few commits on master as a first development project and I keep seeing a huge wall of: "The forked VM terminated without properly saying goodbye".
>
> It also looks like the build has been failing for a few months? Am I seeing that right? http://mail-archives.apache.org/mod_mbox/flink-builds/202101.mbox/browser That's why I think maybe we have a regular practices of skipping some tests or setting some flags or something like that?
>
> It looks like my builds are repeatedly dying in the "Runtime" test suite.
>
> # Specific Failure:
> ```
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]   BlobServerRangeTest.testOnePortAvailable:106
> Expected: (a value equal to or greater than <50000> and a value less than or equal to <50050>)
>       but: a value equal to or greater than <50000> <49877> was less than <50000>
> [ERROR]   BlobServerRangeTest.testPortUnavailable Expected exception: java.io.IOException
> [ERROR]   BootstrapToolsTest.testActorSystemInstantiationFailureWhenPortOccupied:528 Expected to fail with a BindException
> [ERROR]   RestClientTest.testConnectionTimeout:78
> Expected: an instance of org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
>       but: <org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /240.0.0.0:80> is a org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException
> [ERROR]   TaskManagerRunnerStartupTest.testStartupWhenNetworkStackFailsToInitialize:163 Should throw IOException when the network stack cannot be initialized.
> [INFO]
> [ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
> ```
>
> # Stopping at suite: Runtime
> ```
> [ERROR] Tests run: 5072, Failures: 5, Errors: 0, Skipped: 51
> [INFO]
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary for Flink : 1.13-SNAPSHOT:
> [INFO]
> [INFO] Flink : Tools : Force Shading ...................... SUCCESS [ 11.719 s]
> [INFO] Flink : ............................................ SUCCESS [ 22.373 s]
> [INFO] Flink : Annotations ................................ SUCCESS [  4.312 s]
> [INFO] Flink : Test utils : ............................... SUCCESS [  0.384 s]
> [INFO] Flink : Test utils : Junit ......................... SUCCESS [  6.516 s]
> [INFO] Flink : Metrics : .................................. SUCCESS [  0.461 s]
> [INFO] Flink : Metrics : Core ............................. SUCCESS [  4.643 s]
> [INFO] Flink : Core ....................................... SUCCESS [02:00 min]
> [INFO] Flink : Java ....................................... SUCCESS [ 48.244 s]
> [INFO] Flink : Queryable state : .......................... SUCCESS [  0.315 s]
> [INFO] Flink : Queryable state : Client Java .............. SUCCESS [  5.269 s]
> [INFO] Flink : FileSystems : .............................. SUCCESS [  0.461 s]
> [INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [ 17.152 s]
>
>
> [INFO] Flink : Runtime .................................... FAILURE [09:08 min]
>
>
> [INFO] Flink : Scala ...................................... SKIPPED
> [INFO] Flink : FileSystems : Mapr FS ...................... SKIPPED
> [INFO] Flink : FileSystems : Hadoop FS shaded ............. SKIPPED
> [INFO] Flink : FileSystems : S3 FS Base ................... SKIPPED

bytesandwich

Re: First development steps, broken build workarounds?

In reply to this post by bytesandwich

Hi Chesnay and 郁蓝

Thank you for your help. I think Chesnay's feedback makes sense that some
tests are bound to fail under WSL. Maybe if more people encounter this we
can make a marked test suite?

Thanks again,
Jack

--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Chesnay Schepler-3

Re: First development steps, broken build workarounds?

Sure, but for WSL in particular we'd ideally we have way to detect that
we're even running within it. I'm not sure whether there is a sane
solution for that that works across various distros.

On 2/11/2021 1:14 AM, bytesandwich wrote:

> Hi Chesnay and 郁蓝
>
> Thank you for your help. I think Chesnay's feedback makes sense that some
> tests are bound to fail under WSL. Maybe if more people encounter this we
> can make a marked test suite?
>
> Thanks again,
> Jack
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/