ConnectedComponents IT Case

[Sebastian Kunert]

Hey guys,

I am currently working on optimizer integration of forwarded fields. I get
NullPointerExceptions during the execution of our ConnectedComponentITCase
in the NeighborWithComponentIDJoin. A second pair of eyes on this plan
would help me, maybe there are some obvious problems with it that I may
have overlooked:

http://pastebin.com/beBjpSzZ

Thank you :)

Sebastian

[Ufuk Celebi]

Hey Sebastian,

Could you also post the exception?

Thanks!

Ufuk

> On 27 Jul 2014, at 18:23, Sebastian Kunert <[hidden email]> wrote:
>
> Hey guys,
>
> I am currently working on optimizer integration of forwarded fields. I get
> NullPointerExceptions during the execution of our ConnectedComponentITCase
> in the NeighborWithComponentIDJoin. A second pair of eyes on this plan
> would help me, maybe there are some obvious problems with it that I may
> have overlooked:
>
> http://pastebin.com/beBjpSzZ
>
> Thank you :)
>
> Sebastian

[Sebastian Kunert]

Hi,

attached is the whole stacktrace. I am working in this branch
https://github.com/skunert/incubator-flink/tree/constantFields_renamed .The
question is whether the plan is incorrectly build due to my changes or if
there is maybe a optimizer bug which only comes to effect because of my
changes. Thatswhy I am  investigating if the plan itself is correct.

19:56:38,774 ERROR org.apache.flink.runtime.operators.RegularPactTask
     - Error in task code:
 Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
(1/4)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
 at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
 at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
 at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
 at java.lang.Thread.run(Thread.java:745)
19:56:38,802 ERROR org.apache.flink.runtime.operators.RegularPactTask
     - Error in task code:
 Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
(2/4)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
 at org.apache.flink.runtime.operators.JoinWithNullpoint exception when
termination requested was false
Nullpoint exception when termination requested was false
SolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
 at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
 at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
 at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
 at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
 at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
 at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
 at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
 at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
 at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
 at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
 at java.lang.Thread.run(Thread.java:745)
19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
     - Error in task code:
 Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
(4/4)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
 at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
 at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
 at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
 at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
 at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
 at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
 at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
 at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
at java.lang.Thread.run(Thread.java:745)
19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
     - Error in task code:
 Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
(3/4)
java.lang.NullPointerException
 at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
 at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
 at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
 at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
 at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
 at
org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
at
org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
 at
org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
at
org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
 at
org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
 at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
at java.lang.Thread.run(Thread.java:745)
Nullpoint exception when termination requested was false
Nullpoint exception when termination requested was false

java.lang.AssertionError: Vertex is in wrong component.
Expected :2
Actual   :514
 <Click to see difference>
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
 at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at
org.apache.flink.test.testdata.ConnectedComponentsData.checkOddEvenResult(ConnectedComponentsData.java:104)
 at
org.apache.flink.test.exampleJavaPrograms.ConnectedComponentsITCase.postSubmit(ConnectedComponentsITCase.java:57)
at
org.apache.flink.test.util.JavaProgramTestBase.testJob(JavaProgramTestBase.java:113)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
 at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
 at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)


2014-07-27 20:06 GMT+02:00 Ufuk Celebi <[hidden email]>:

> Hey Sebastian,
>
> Could you also post the exception?
>
> Thanks!
>
> Ufuk
>
> > On 27 Jul 2014, at 18:23, Sebastian Kunert <[hidden email]> wrote:
> >
> > Hey guys,
> >
> > I am currently working on optimizer integration of forwarded fields. I
> get
> > NullPointerExceptions during the execution of our
> ConnectedComponentITCase
> > in the NeighborWithComponentIDJoin. A second pair of eyes on this plan
> > would help me, maybe there are some obvious problems with it that I may
> > have overlooked:
> >
> > http://pastebin.com/beBjpSzZ
> >
> > Thank you :)
> >
> > Sebastian
>

[Ufuk Celebi]

Thanks! I didn't take a look at the plan or code yet, but the call of joinFunction.join(probeSideRecord, null, collector) in JoinWithSolutionSetSecondDriver.java:143 is the root of the problem. It is taking the branch, because the solution set join is *not* finding a match in the hash table for some record.

Did you specifically look into the hash table related parts while debugging? This is something I would do, since actually the initial solution set should ensure that there is always a matching record in the hash table.

If this is not of much help, we can also do a debugging session via screen sharing tomorrow.

Ufuk

On 27 Jul 2014, at 20:17, Sebastian Kunert <[hidden email]> wrote:

> Hi,
>
> attached is the whole stacktrace. I am working in this branch
> https://github.com/skunert/incubator-flink/tree/constantFields_renamed .The
> question is whether the plan is incorrectly build due to my changes or if
> there is maybe a optimizer bug which only comes to effect because of my
> changes. Thatswhy I am  investigating if the plan itself is correct.
>
> 19:56:38,774 ERROR org.apache.flink.runtime.operators.RegularPactTask
>     - Error in task code:
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> (1/4)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> 19:56:38,802 ERROR org.apache.flink.runtime.operators.RegularPactTask
>     - Error in task code:
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> (2/4)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at org.apache.flink.runtime.operators.JoinWithNullpoint exception when
> termination requested was false
> Nullpoint exception when termination requested was false
> SolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> 19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
>     - Error in task code:
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> (4/4)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> 19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
>     - Error in task code:
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> (3/4)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> java.lang.NullPointerException
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> at
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> at
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> at
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> at
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> at java.lang.Thread.run(Thread.java:745)
> Nullpoint exception when termination requested was false
> Nullpoint exception when termination requested was false
>
> java.lang.AssertionError: Vertex is in wrong component.
> Expected :2
> Actual   :514
> <Click to see difference>
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.failNotEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:126)
> at org.junit.Assert.assertEquals(Assert.java:470)
> at
> org.apache.flink.test.testdata.ConnectedComponentsData.checkOddEvenResult(ConnectedComponentsData.java:104)
> at
> org.apache.flink.test.exampleJavaPrograms.ConnectedComponentsITCase.postSubmit(ConnectedComponentsITCase.java:57)
> at
> org.apache.flink.test.util.JavaProgramTestBase.testJob(JavaProgramTestBase.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>
>
> 2014-07-27 20:06 GMT+02:00 Ufuk Celebi <[hidden email]>:
>
>> Hey Sebastian,
>>
>> Could you also post the exception?
>>
>> Thanks!
>>
>> Ufuk
>>
>>> On 27 Jul 2014, at 18:23, Sebastian Kunert <[hidden email]> wrote:
>>>
>>> Hey guys,
>>>
>>> I am currently working on optimizer integration of forwarded fields. I
>> get
>>> NullPointerExceptions during the execution of our
>> ConnectedComponentITCase
>>> in the NeighborWithComponentIDJoin. A second pair of eyes on this plan
>>> would help me, maybe there are some obvious problems with it that I may
>>> have overlooked:
>>>
>>> http://pastebin.com/beBjpSzZ
>>>
>>> Thank you :)
>>>
>>> Sebastian
>>

[Sebastian Kunert]

Hey Ufuk,

thank you for this. I have not yet taken a look at this hash table, will
look into it tommorrow at the office.

Sebastian


2014-07-27 21:02 GMT+02:00 Ufuk Celebi <[hidden email]>:

> Thanks! I didn't take a look at the plan or code yet, but the call of
> joinFunction.join(probeSideRecord, null, collector) in
> JoinWithSolutionSetSecondDriver.java:143 is the root of the problem. It is
> taking the branch, because the solution set join is *not* finding a match
> in the hash table for some record.
>
> Did you specifically look into the hash table related parts while
> debugging? This is something I would do, since actually the initial
> solution set should ensure that there is always a matching record in the
> hash table.
>
> If this is not of much help, we can also do a debugging session via screen
> sharing tomorrow.
>
> Ufuk
>
> On 27 Jul 2014, at 20:17, Sebastian Kunert <[hidden email]> wrote:
>
> > Hi,
> >
> > attached is the whole stacktrace. I am working in this branch
> > https://github.com/skunert/incubator-flink/tree/constantFields_renamed
> .The
> > question is whether the plan is incorrectly build due to my changes or if
> > there is maybe a optimizer bug which only comes to effect because of my
> > changes. Thatswhy I am  investigating if the plan itself is correct.
> >
> > 19:56:38,774 ERROR org.apache.flink.runtime.operators.RegularPactTask
> >     - Error in task code:
> >
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> > (1/4)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > 19:56:38,802 ERROR org.apache.flink.runtime.operators.RegularPactTask
> >     - Error in task code:
> >
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> > (2/4)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at org.apache.flink.runtime.operators.JoinWithNullpoint exception when
> > termination requested was false
> > Nullpoint exception when termination requested was false
> > SolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > 19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
> >     - Error in task code:
> >
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> > (4/4)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > 19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
> >     - Error in task code:
> >
> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
> > (3/4)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > java.lang.NullPointerException
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
> > at
> >
> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
> > at
> >
> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
> > at
> >
> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
> > at
> >
> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
> > at
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
> > at
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
> > at java.lang.Thread.run(Thread.java:745)
> > Nullpoint exception when termination requested was false
> > Nullpoint exception when termination requested was false
> >
> > java.lang.AssertionError: Vertex is in wrong component.
> > Expected :2
> > Actual   :514
> > <Click to see difference>
> > at org.junit.Assert.fail(Assert.java:91)
> > at org.junit.Assert.failNotEquals(Assert.java:645)
> > at org.junit.Assert.assertEquals(Assert.java:126)
> > at org.junit.Assert.assertEquals(Assert.java:470)
> > at
> >
> org.apache.flink.test.testdata.ConnectedComponentsData.checkOddEvenResult(ConnectedComponentsData.java:104)
> > at
> >
> org.apache.flink.test.exampleJavaPrograms.ConnectedComponentsITCase.postSubmit(ConnectedComponentsITCase.java:57)
> > at
> >
> org.apache.flink.test.util.JavaProgramTestBase.testJob(JavaProgramTestBase.java:113)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> > at
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> > at
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> > at
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> > at
> >
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> > at
> >
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> > at
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> > at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
> > at
> >
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
> > at
> >
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
> > at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> >
> >
> > 2014-07-27 20:06 GMT+02:00 Ufuk Celebi <[hidden email]>:
> >
> >> Hey Sebastian,
> >>
> >> Could you also post the exception?
> >>
> >> Thanks!
> >>
> >> Ufuk
> >>
> >>> On 27 Jul 2014, at 18:23, Sebastian Kunert <[hidden email]>
> wrote:
> >>>
> >>> Hey guys,
> >>>
> >>> I am currently working on optimizer integration of forwarded fields. I
> >> get
> >>> NullPointerExceptions during the execution of our
> >> ConnectedComponentITCase
> >>> in the NeighborWithComponentIDJoin. A second pair of eyes on this plan
> >>> would help me, maybe there are some obvious problems with it that I may
> >>> have overlooked:
> >>>
> >>> http://pastebin.com/beBjpSzZ
> >>>
> >>> Thank you :)
> >>>
> >>> Sebastian
> >>
>
>

[Sebastian Kunert]

Hey,

the problem is now fixed. But during investigation, i stubled upon these
lines in JoinWithSolutionSetSecondDriver.java (lines 139 ff).

while (this.running && ((probeSideRecord =
probeSideInput.next(probeSideRecord)) != null)) {
 if (prober.getMatchFor(probeSideRecord, buildSideRecord)) {
joinFunction.join(probeSideRecord, buildSideRecord, collector);
 } else {
joinFunction.join(probeSideRecord, null, collector);  // <-- strange null
value here
 }
}

Is it intentional that a null value is handed to a user function here? This
looks like a bug to me.


2014-07-27 21:09 GMT+02:00 Sebastian Kunert <[hidden email]>:

> Hey Ufuk,
>
> thank you for this. I have not yet taken a look at this hash table, will
> look into it tommorrow at the office.
>
> Sebastian
>
>
> 2014-07-27 21:02 GMT+02:00 Ufuk Celebi <[hidden email]>:
>
> Thanks! I didn't take a look at the plan or code yet, but the call of
>> joinFunction.join(probeSideRecord, null, collector) in
>> JoinWithSolutionSetSecondDriver.java:143 is the root of the problem. It is
>> taking the branch, because the solution set join is *not* finding a match
>> in the hash table for some record.
>>
>> Did you specifically look into the hash table related parts while
>> debugging? This is something I would do, since actually the initial
>> solution set should ensure that there is always a matching record in the
>> hash table.
>>
>> If this is not of much help, we can also do a debugging session via
>> screen sharing tomorrow.
>>
>> Ufuk
>>
>> On 27 Jul 2014, at 20:17, Sebastian Kunert <[hidden email]> wrote:
>>
>> > Hi,
>> >
>> > attached is the whole stacktrace. I am working in this branch
>> > https://github.com/skunert/incubator-flink/tree/constantFields_renamed
>> .The
>> > question is whether the plan is incorrectly build due to my changes or
>> if
>> > there is maybe a optimizer bug which only comes to effect because of my
>> > changes. Thatswhy I am  investigating if the plan itself is correct.
>> >
>> > 19:56:38,774 ERROR org.apache.flink.runtime.operators.RegularPactTask
>> >     - Error in task code:
>> >
>> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
>> > (1/4)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > 19:56:38,802 ERROR org.apache.flink.runtime.operators.RegularPactTask
>> >     - Error in task code:
>> >
>> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
>> > (2/4)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at org.apache.flink.runtime.operators.JoinWithNullpoint exception when
>> > termination requested was false
>> > Nullpoint exception when termination requested was false
>> > SolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > 19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
>> >     - Error in task code:
>> >
>> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
>> > (4/4)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > 19:56:38,831 ERROR org.apache.flink.runtime.operators.RegularPactTask
>> >     - Error in task code:
>> >
>> Join(org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter)
>> > (3/4)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > java.lang.NullPointerException
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:176)
>> > at
>> >
>> org.apache.flink.example.java.graph.ConnectedComponents$ComponentIdFilter.join(ConnectedComponents.java:171)
>> > at
>> >
>> org.apache.flink.runtime.operators.JoinWithSolutionSetSecondDriver.run(JoinWithSolutionSetSecondDriver.java:143)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:509)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:137)
>> > at
>> >
>> org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:106)
>> > at
>> >
>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:374)
>> > at
>> >
>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Nullpoint exception when termination requested was false
>> > Nullpoint exception when termination requested was false
>> >
>> > java.lang.AssertionError: Vertex is in wrong component.
>> > Expected :2
>> > Actual   :514
>> > <Click to see difference>
>> > at org.junit.Assert.fail(Assert.java:91)
>> > at org.junit.Assert.failNotEquals(Assert.java:645)
>> > at org.junit.Assert.assertEquals(Assert.java:126)
>> > at org.junit.Assert.assertEquals(Assert.java:470)
>> > at
>> >
>> org.apache.flink.test.testdata.ConnectedComponentsData.checkOddEvenResult(ConnectedComponentsData.java:104)
>> > at
>> >
>> org.apache.flink.test.exampleJavaPrograms.ConnectedComponentsITCase.postSubmit(ConnectedComponentsITCase.java:57)
>> > at
>> >
>> org.apache.flink.test.util.JavaProgramTestBase.testJob(JavaProgramTestBase.java:113)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>> > at
>> >
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> > at
>> >
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>> > at
>> >
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>> > at
>> >
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>> > at
>> >
>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
>> > at
>> >
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
>> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
>> > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
>> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
>> > at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>> > at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
>> > at
>> >
>> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
>> > at
>> >
>> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
>> > at
>> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>> >
>> >
>> > 2014-07-27 20:06 GMT+02:00 Ufuk Celebi <[hidden email]>:
>> >
>> >> Hey Sebastian,
>> >>
>> >> Could you also post the exception?
>> >>
>> >> Thanks!
>> >>
>> >> Ufuk
>> >>
>> >>> On 27 Jul 2014, at 18:23, Sebastian Kunert <[hidden email]>
>> wrote:
>> >>>
>> >>> Hey guys,
>> >>>
>> >>> I am currently working on optimizer integration of forwarded fields. I
>> >> get
>> >>> NullPointerExceptions during the execution of our
>> >> ConnectedComponentITCase
>> >>> in the NeighborWithComponentIDJoin. A second pair of eyes on this plan
>> >>> would help me, maybe there are some obvious problems with it that I
>> may
>> >>> have overlooked:
>> >>>
>> >>> http://pastebin.com/beBjpSzZ
>> >>>
>> >>> Thank you :)
>> >>>
>> >>> Sebastian
>> >>
>>
>>
>

[Ufuk Celebi]

On 29 Jul 2014, at 15:32, Sebastian Kunert <[hidden email]> wrote:
> Is it intentional that a null value is handed to a user function here? This
> looks like a bug to me.

I think this is intentional for solution set joins. The problem is that it changes the join semantics to become an outer join.

We should be clear that solution set joins can return null if there is no match in the table.

For the near future, I would vote to provide a further join methods like outer join (also for non-iterative programs).

[Stephan Ewen]

Ufuk is right, that is intentional right now.

If you get a null and do not expect it, check the program. Or in your case,
the plan from the optimizer ;-)

[Fabian Hueske]

OK, let me summarize.

Ufuk says, that is correct because a user might want to insert a new
element into the solution set and therefore should check for null in the
UDF.
And Stephan adds that a user should check the plan if a null is passed to
the UDF unexpectedly (which was the reason in our case due to incorrect
annoations).

Is this behavior documented somewhere?
Does it make sense to use the regular API join method for this or should we
rather make the join with a solution set more explicit?
Can we "force" the user to do

iteration.getSolutionSet().joinWithSolutionSet(myData)

instead of

myData.join(iteration.getSolutionSet())?

In that case we could make this behavior more explicit by using another
method name and could also prevent that a user calls any other method than
join or coGroup on a solution set which is prohibited right now.


2014-07-29 15:58 GMT+02:00 Stephan Ewen <[hidden email]>:

> Ufuk is right, that is intentional right now.
>
> If you get a null and do not expect it, check the program. Or in your case,
> the plan from the optimizer ;-)
>

[Ufuk Celebi]

On 29 Jul 2014, at 17:13, Fabian Hueske <[hidden email]> wrote:

> OK, let me summarize.
>
> Ufuk says, that is correct because a user might want to insert a new
> element into the solution set and therefore should check for null in the
> UDF.

I actually said/wanted to say that it is *not* correct, because it breaks the join semantics, but that it is useful for the insertion use case.

> And Stephan adds that a user should check the plan if a null is passed to
> the UDF unexpectedly (which was the reason in our case due to incorrect
> annoations).

I think Stephan explicitly referred to Sebastian and wouldn't expect a normal user to look at the plan. ;-)

> Is this behavior documented somewhere?

If not, we should definitely add it.

> Does it make sense to use the regular API join method for this or should we
> rather make the join with a solution set more explicit?
> Can we "force" the user to do
>
> iteration.getSolutionSet().joinWithSolutionSet(myData)
>
> instead of
>
> myData.join(iteration.getSolutionSet())?
>
> In that case we could make this behavior more explicit by using another
> method name and could also prevent that a user calls any other method than
> join or coGroup on a solution set which is prohibited right now.

+1 I agree that we should make this more explicit.

If we added further joins we wouldn't need an extra `joinWithSolutionSet`, but the user would pick the one she wants.

[Stephan Ewen]

I agree with Ufuk.

A high priority for the next release should be to have outer join
functionality explicitly.

Stephan

[Fabian Hueske-2]

Alright, thanks for clarifiying. (Sorry Ufuk for misquoting... ;-) )

What would be the strategy?

1) Do we want to support the "null outer join" and add documentation or
drop this feature for now and throw an exception?
2) If we want to keep the functionality, does it make sense to change the
API, in the way I proposed (explicit joins with the solution set) or should
we keep it as it is and later simply add outer join functionality? The
latter would treat a join with a solution set as any other join while the
first one "knows" that it is special and would also restrict the operations
on the solution set.


2014-07-29 17:38 GMT+02:00 Stephan Ewen <[hidden email]>:

> I agree with Ufuk.
>
> A high priority for the next release should be to have outer join
> functionality explicitly.
>
> Stephan
>