FileNotFoundException thrown by BlobCache when running "mvn test" against flink-runtime 0.10 for Scala 2.11

[Prez Cannady]

Steps

Forked, cut bug fix branch off of release–0.10.
One line change to fix class loader issue.
Ran mvn clean test in flink-runtime. All 894 tests pass.
Ran ./tools/change-scala-version 2.11 in the root.
Ran mvn clean test -Dscala.version=2.11.7 in flink-runtime.
Expected result

All 894 tests pass. without incident

Actual result

All 894 tests actually did pass, but frequently seeing the exception below in the log (with context, and also available as a gist:

Discussion: Not sure if this is an issue, but figured I’d ask before submitting my pull request.

6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache                       - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157 (retry 5)
6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient                      - GET content addressable BLOB e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache                       - Failed to fetch BLOB e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157 and store it under /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4 No retries left.
java.io.FileNotFoundException: /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4 (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
        at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
        at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
        at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
        at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

Prez Cannady  
p: 617 500 3378  
e: [hidden email] <mailto:[hidden email]>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  

[Prez Cannady]

My bad.  Gist with exception is here:

https://gist.github.com/revprez/5a730a45674f0fc6e52b <https://gist.github.com/revprez/5a730a45674f0fc6e52b>

Prez Cannady  
p: 617 500 3378  
e: [hidden email] <mailto:[hidden email]>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  

> On Jan 14, 2016, at 7:16 PM, Prez Cannady <[hidden email]> wrote:
>
> Steps
>
> Forked, cut bug fix branch off of release–0.10.
> One line change to fix class loader issue.
> Ran mvn clean test in flink-runtime. All 894 tests pass.
> Ran ./tools/change-scala-version 2.11 in the root.
> Ran mvn clean test -Dscala.version=2.11.7 in flink-runtime.
> Expected result
>
> All 894 tests pass. without incident
>
> Actual result
>
> All 894 tests actually did pass, but frequently seeing the exception below in the log (with context, and also available as a gist:
>
> Discussion: Not sure if this is an issue, but figured I’d ask before submitting my pull request.
>
> 6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache                       - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157 (retry 5)
> 6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient                      - GET content addressable BLOB e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
> 6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache                       - Failed to fetch BLOB e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157 and store it under /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4 No retries left.
> java.io.FileNotFoundException: /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4 (Permission denied)
>         at java.io.FileOutputStream.open0(Native Method)
>         at java.io.FileOutputStream.open(FileOutputStream.java:270)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
>         at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
>         at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
>         at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
>         at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
>         at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>         at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>         at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>
> Prez Cannady  
> p: 617 500 3378  
> e: [hidden email] <mailto:[hidden email]>  
> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
> LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  
>
>
>
>
>
>
>
>
>

[Robert Metzger]

I think the exceptions are fine. We have many tests which also check
certain error conditions. I guess the exceptions are expected by the tests.

On Fri, Jan 15, 2016 at 1:20 AM, Prez Cannady <[hidden email]>
wrote:

> My bad.  Gist with exception is here:
>
> https://gist.github.com/revprez/5a730a45674f0fc6e52b <
> https://gist.github.com/revprez/5a730a45674f0fc6e52b>
>
> Prez Cannady
> p: 617 500 3378
> e: [hidden email] <mailto:[hidden email]>
> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
> LI: https://www.linkedin.com/in/revprez <
> https://www.linkedin.com/in/revprez>
>
>
>
>
>
>
>
>
>
> > On Jan 14, 2016, at 7:16 PM, Prez Cannady <[hidden email]>
> wrote:
> >
> > Steps
> >
> > Forked, cut bug fix branch off of release–0.10.
> > One line change to fix class loader issue.
> > Ran mvn clean test in flink-runtime. All 894 tests pass.
> > Ran ./tools/change-scala-version 2.11 in the root.
> > Ran mvn clean test -Dscala.version=2.11.7 in flink-runtime.
> > Expected result
> >
> > All 894 tests pass. without incident
> >
> > Actual result
> >
> > All 894 tests actually did pass, but frequently seeing the exception
> below in the log (with context, and also available as a gist:
> >
> > Discussion: Not sure if this is an issue, but figured I’d ask before
> submitting my pull request.
> >
> > 6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache
>              - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from
> localhost/127.0.0.1:61157 (retry 5)
> > 6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient
>             - GET content addressable BLOB
> e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
> > 6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache
>              - Failed to fetch BLOB
> e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157
> and store it under
> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
> No retries left.
> > java.io.FileNotFoundException:
> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
> (Permission denied)
> >         at java.io.FileOutputStream.open0(Native Method)
> >         at java.io.FileOutputStream.open(FileOutputStream.java:270)
> >         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
> >         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
> >         at
> org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
> >         at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
> >         at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
> >         at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:497)
> >         at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> >         at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> >         at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> >         at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> >         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> >         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> >         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> >         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> >         at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> >         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> >         at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> >         at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> >         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> >         at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> >         at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> >         at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> >         at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> >         at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> >         at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> >         at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> >
> > Prez Cannady
> > p: 617 500 3378
> > e: [hidden email] <mailto:[hidden email]>
> > GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
> > LI: https://www.linkedin.com/in/revprez <
> https://www.linkedin.com/in/revprez>
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>

[Stephan Ewen]

Hi!

I cannot access the gist.

There are some tests that check proper error reporting in that case.
Let's make sure there is nothing funky in the sense that this occurs also
outside where this is tested for and goes unrecognized by the tests.

Can you share more of the log?

Thanks,
Stephan


On Fri, Jan 15, 2016 at 9:52 AM, Robert Metzger <[hidden email]> wrote:

> I think the exceptions are fine. We have many tests which also check
> certain error conditions. I guess the exceptions are expected by the tests.
>
> On Fri, Jan 15, 2016 at 1:20 AM, Prez Cannady <[hidden email]>
> wrote:
>
> > My bad.  Gist with exception is here:
> >
> > https://gist.github.com/revprez/5a730a45674f0fc6e52b <
> > https://gist.github.com/revprez/5a730a45674f0fc6e52b>
> >
> > Prez Cannady
> > p: 617 500 3378
> > e: [hidden email] <mailto:[hidden email]>
> > GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
> > LI: https://www.linkedin.com/in/revprez <
> > https://www.linkedin.com/in/revprez>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > > On Jan 14, 2016, at 7:16 PM, Prez Cannady <[hidden email]>
> > wrote:
> > >
> > > Steps
> > >
> > > Forked, cut bug fix branch off of release–0.10.
> > > One line change to fix class loader issue.
> > > Ran mvn clean test in flink-runtime. All 894 tests pass.
> > > Ran ./tools/change-scala-version 2.11 in the root.
> > > Ran mvn clean test -Dscala.version=2.11.7 in flink-runtime.
> > > Expected result
> > >
> > > All 894 tests pass. without incident
> > >
> > > Actual result
> > >
> > > All 894 tests actually did pass, but frequently seeing the exception
> > below in the log (with context, and also available as a gist:
> > >
> > > Discussion: Not sure if this is an issue, but figured I’d ask before
> > submitting my pull request.
> > >
> > > 6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache
> >              - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from
> > localhost/127.0.0.1:61157 (retry 5)
> > > 6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient
> >             - GET content addressable BLOB
> > e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
> > > 6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache
> >              - Failed to fetch BLOB
> > e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157
> > and store it under
> >
> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
> > No retries left.
> > > java.io.FileNotFoundException:
> >
> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
> > (Permission denied)
> > >         at java.io.FileOutputStream.open0(Native Method)
> > >         at java.io.FileOutputStream.open(FileOutputStream.java:270)
> > >         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
> > >         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
> > >         at
> > org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
> > >         at
> >
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
> > >         at
> >
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
> > >         at
> >
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
> > >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >         at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >         at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >         at java.lang.reflect.Method.invoke(Method.java:497)
> > >         at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> > >         at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > >         at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> > >         at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > >         at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> > >         at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> > >         at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> > >         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> > >         at
> > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> > >         at
> > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> > >         at
> > org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> > >         at
> > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> > >         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> > >         at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> > >         at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> > >         at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> > >         at
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> > >         at
> >
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> > >         at
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> > >         at
> > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> > >
> > > Prez Cannady
> > > p: 617 500 3378
> > > e: [hidden email] <mailto:[hidden email]>
> > > GH: https://github.com/opencorrelate <https://github.com/opencorrelate
> >
> > > LI: https://www.linkedin.com/in/revprez <
> > https://www.linkedin.com/in/revprez>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>

[Prez Cannady]

Recreated it here.

https://gist.github.com/revprez/2e5d7b38a1b5fe1d704f <https://gist.github.com/revprez/2e5d7b38a1b5fe1d704f>

Prez Cannady  
p: 617 500 3378  
e: [hidden email] <mailto:[hidden email]>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  

> On Jan 14, 2016, at 7:16 PM, Prez Cannady <[hidden email]> wrote:
>
> 6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache                       - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157 (retry 5)
> 6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient                      - GET content addressable BLOB e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
> 6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache                       - Failed to fetch BLOB e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157 and store it under /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4 No retries left.
> java.io.FileNotFoundException: /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4 (Permission denied)
>         at java.io.FileOutputStream.open0(Native Method)
>         at java.io.FileOutputStream.open(FileOutputStream.java:270)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
>         at org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
>         at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
>         at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
>         at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>         at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>         at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
>         at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>         at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>         at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

[Prez Cannady-2]

I’ve included the full log of a test run against flink-runtime here:

https://gist.github.com/revprez/9d38749f6672d1bef9f1 <https://gist.github.com/revprez/9d38749f6672d1bef9f1>

Specific issue observed starts at 691, but you’ll see exceptions both before and after.  As per Metzger's observation, I think BlobCache is simply testing for some previous BlobServer and after its retries run out a new BlobServer is created to handle requests.  All the FileNotFoundExceptions occur within a cluster of the test run and do not emerge again.  I’m not clear, however, which test is associated with these printouts.  The stack trace doesn’t go any higher than org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103).

There are other exceptions

Prez Cannady  
p: 617 500 3378  
e: [hidden email] <mailto:[hidden email]>  
GH: https://github.com/opencorrelate <https://github.com/opencorrelate>  
LI: https://www.linkedin.com/in/revprez <https://www.linkedin.com/in/revprez>  

> On Jan 15, 2016, at 6:28 AM, Stephan Ewen <[hidden email]> wrote:
>
> Hi!
>
> I cannot access the gist.
>
> There are some tests that check proper error reporting in that case.
> Let's make sure there is nothing funky in the sense that this occurs also
> outside where this is tested for and goes unrecognized by the tests.
>
> Can you share more of the log?
>
> Thanks,
> Stephan
>
>
> On Fri, Jan 15, 2016 at 9:52 AM, Robert Metzger <[hidden email]> wrote:
>
>> I think the exceptions are fine. We have many tests which also check
>> certain error conditions. I guess the exceptions are expected by the tests.
>>
>> On Fri, Jan 15, 2016 at 1:20 AM, Prez Cannady <[hidden email]>
>> wrote:
>>
>>> My bad.  Gist with exception is here:
>>>
>>> https://gist.github.com/revprez/5a730a45674f0fc6e52b <
>>> https://gist.github.com/revprez/5a730a45674f0fc6e52b>
>>>
>>> Prez Cannady
>>> p: 617 500 3378
>>> e: [hidden email] <mailto:[hidden email]>
>>> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
>>> LI: https://www.linkedin.com/in/revprez <
>>> https://www.linkedin.com/in/revprez>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On Jan 14, 2016, at 7:16 PM, Prez Cannady <[hidden email]>
>>> wrote:
>>>>
>>>> Steps
>>>>
>>>> Forked, cut bug fix branch off of release–0.10.
>>>> One line change to fix class loader issue.
>>>> Ran mvn clean test in flink-runtime. All 894 tests pass.
>>>> Ran ./tools/change-scala-version 2.11 in the root.
>>>> Ran mvn clean test -Dscala.version=2.11.7 in flink-runtime.
>>>> Expected result
>>>>
>>>> All 894 tests pass. without incident
>>>>
>>>> Actual result
>>>>
>>>> All 894 tests actually did pass, but frequently seeing the exception
>>> below in the log (with context, and also available as a gist:
>>>>
>>>> Discussion: Not sure if this is an issue, but figured I’d ask before
>>> submitting my pull request.
>>>>
>>>> 6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache
>>>             - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from
>>> localhost/127.0.0.1:61157 (retry 5)
>>>> 6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient
>>>            - GET content addressable BLOB
>>> e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
>>>> 6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache
>>>             - Failed to fetch BLOB
>>> e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/127.0.0.1:61157
>>> and store it under
>>>
>> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
>>> No retries left.
>>>> java.io.FileNotFoundException:
>>>
>> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
>>> (Permission denied)
>>>>        at java.io.FileOutputStream.open0(Native Method)
>>>>        at java.io.FileOutputStream.open(FileOutputStream.java:270)
>>>>        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
>>>>        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
>>>>        at
>>> org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
>>>>        at
>>>
>> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
>>>>        at
>>>
>> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
>>>>        at
>>>
>> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
>>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>        at
>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>        at
>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>        at java.lang.reflect.Method.invoke(Method.java:497)
>>>>        at
>>>
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>>>>        at
>>>
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>>>>        at
>>>
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>>>>        at
>>>
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>>>>        at
>> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>>>>        at
>>>
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>>>>        at
>>>
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>>>>        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>>>>        at
>>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>>>>        at
>>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>>>>        at
>>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>>>>        at
>>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>>>>        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>>>>        at
>>>
>> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
>>>>        at
>>>
>> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
>>>>        at
>>>
>> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>>>>        at
>>>
>> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
>>>>        at
>>>
>> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>>>>        at
>>>
>> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>>>>        at
>>> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>>>>
>>>> Prez Cannady
>>>> p: 617 500 3378
>>>> e: [hidden email] <mailto:[hidden email]>
>>>> GH: https://github.com/opencorrelate <https://github.com/opencorrelate
>>>
>>>> LI: https://www.linkedin.com/in/revprez <
>>> https://www.linkedin.com/in/revprez>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>

[Till Rohrmann]

A problem is that all the BlobCache/Server tests don't extend the
`TestLogger` class which logs the names of the different tests. Maybe we
should that add as well.

On Fri, Jan 15, 2016 at 3:32 PM, Prez Cannady <[hidden email]>
wrote:

> I’ve included the full log of a test run against flink-runtime here:
>
> https://gist.github.com/revprez/9d38749f6672d1bef9f1 <
> https://gist.github.com/revprez/9d38749f6672d1bef9f1>
>
> Specific issue observed starts at 691, but you’ll see exceptions both
> before and after.  As per Metzger's observation, I think BlobCache is
> simply testing for some previous BlobServer and after its retries run out a
> new BlobServer is created to handle requests.  All the
> FileNotFoundExceptions occur within a cluster of the test run and do not
> emerge again.  I’m not clear, however, which test is associated with these
> printouts.  The stack trace doesn’t go any higher than
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103).
>
> There are other exceptions
>
> Prez Cannady
> p: 617 500 3378
> e: [hidden email] <mailto:[hidden email]>
> GH: https://github.com/opencorrelate <https://github.com/opencorrelate>
> LI: https://www.linkedin.com/in/revprez <
> https://www.linkedin.com/in/revprez>
>
>
>
>
>
>
>
>
>
> > On Jan 15, 2016, at 6:28 AM, Stephan Ewen <[hidden email]> wrote:
> >
> > Hi!
> >
> > I cannot access the gist.
> >
> > There are some tests that check proper error reporting in that case.
> > Let's make sure there is nothing funky in the sense that this occurs also
> > outside where this is tested for and goes unrecognized by the tests.
> >
> > Can you share more of the log?
> >
> > Thanks,
> > Stephan
> >
> >
> > On Fri, Jan 15, 2016 at 9:52 AM, Robert Metzger <[hidden email]>
> wrote:
> >
> >> I think the exceptions are fine. We have many tests which also check
> >> certain error conditions. I guess the exceptions are expected by the
> tests.
> >>
> >> On Fri, Jan 15, 2016 at 1:20 AM, Prez Cannady <
> [hidden email]>
> >> wrote:
> >>
> >>> My bad.  Gist with exception is here:
> >>>
> >>> https://gist.github.com/revprez/5a730a45674f0fc6e52b <
> >>> https://gist.github.com/revprez/5a730a45674f0fc6e52b>
> >>>
> >>> Prez Cannady
> >>> p: 617 500 3378
> >>> e: [hidden email] <mailto:[hidden email]>
> >>> GH: https://github.com/opencorrelate <https://github.com/opencorrelate
> >
> >>> LI: https://www.linkedin.com/in/revprez <
> >>> https://www.linkedin.com/in/revprez>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> On Jan 14, 2016, at 7:16 PM, Prez Cannady <[hidden email]>
> >>> wrote:
> >>>>
> >>>> Steps
> >>>>
> >>>> Forked, cut bug fix branch off of release–0.10.
> >>>> One line change to fix class loader issue.
> >>>> Ran mvn clean test in flink-runtime. All 894 tests pass.
> >>>> Ran ./tools/change-scala-version 2.11 in the root.
> >>>> Ran mvn clean test -Dscala.version=2.11.7 in flink-runtime.
> >>>> Expected result
> >>>>
> >>>> All 894 tests pass. without incident
> >>>>
> >>>> Actual result
> >>>>
> >>>> All 894 tests actually did pass, but frequently seeing the exception
> >>> below in the log (with context, and also available as a gist:
> >>>>
> >>>> Discussion: Not sure if this is an issue, but figured I’d ask before
> >>> submitting my pull request.
> >>>>
> >>>> 6 > 19:06:15,250 INFO  org.apache.flink.runtime.blob.BlobCache
> >>>             - Downloading e52e6f94e5afa6502299cf982d31105111e1d9a4 from
> >>> localhost/127.0.0.1:61157 (retry 5)
> >>>> 6 > 19:06:15,251 DEBUG org.apache.flink.runtime.blob.BlobClient
> >>>            - GET content addressable BLOB
> >>> e52e6f94e5afa6502299cf982d31105111e1d9a4 from /127.0.0.1:61170
> >>>> 6 > 19:06:15,251 ERROR org.apache.flink.runtime.blob.BlobCache
> >>>             - Failed to fetch BLOB
> >>> e52e6f94e5afa6502299cf982d31105111e1d9a4 from localhost/
> 127.0.0.1:61157
> >>> and store it under
> >>>
> >>
> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
> >>> No retries left.
> >>>> java.io.FileNotFoundException:
> >>>
> >>
> /var/folders/68/82g3zl512nj15b82m5f4nhb40000gn/T/blobStore-dee0aeaa-8199-4024-ba44-4e39154aa8e8/cache/blob_e52e6f94e5afa6502299cf982d31105111e1d9a4
> >>> (Permission denied)
> >>>>        at java.io.FileOutputStream.open0(Native Method)
> >>>>        at java.io.FileOutputStream.open(FileOutputStream.java:270)
> >>>>        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
> >>>>        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
> >>>>        at
> >>> org.apache.flink.runtime.blob.BlobCache.getURL(BlobCache.java:126)
> >>>>        at
> >>>
> >>
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerReferenceToBlobKeyAndGetURL(BlobLibraryCacheManager.java:245)
> >>>>        at
> >>>
> >>
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:114)
> >>>>        at
> >>>
> >>
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManagerTest.testRegisterAndDownload(BlobLibraryCacheManagerTest.java:209)
> >>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>        at
> >>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>>        at
> >>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>        at java.lang.reflect.Method.invoke(Method.java:497)
> >>>>        at
> >>>
> >>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> >>>>        at
> >>>
> >>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> >>>>        at
> >>>
> >>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> >>>>        at
> >>>
> >>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> >>>>        at
> >> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> >>>>        at
> >>>
> >>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> >>>>        at
> >>>
> >>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> >>>>        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> >>>>        at
> >>> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> >>>>        at
> >>> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> >>>>        at
> >>> org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> >>>>        at
> >>> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> >>>>        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> >>>>        at
> >>>
> >>
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> >>>>        at
> >>>
> >>
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> >>>>        at
> >>>
> >>
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> >>>>        at
> >>>
> >>
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> >>>>        at
> >>>
> >>
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> >>>>        at
> >>>
> >>
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> >>>>        at
> >>>
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> >>>>
> >>>> Prez Cannady
> >>>> p: 617 500 3378
> >>>> e: [hidden email] <mailto:[hidden email]>
> >>>> GH: https://github.com/opencorrelate <
> https://github.com/opencorrelate
> >>>
> >>>> LI: https://www.linkedin.com/in/revprez <
> >>> https://www.linkedin.com/in/revprez>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
>
>