(DEPRECATED) Apache Flink Mailing List archive.

TopSpeedWindowing - in error: Could not forward element to next operator

classic Classic list List threaded Threaded
♦
♦
 5 messages Options
Ovidiu Cristian Marcu
Reply | Threaded
Open this post in threaded view
♦
♦
|

TopSpeedWindowing - in error: Could not forward element to next operator

Ovidiu Cristian Marcu
Could you check the following issue on master?

When running this example org.apache.flink.streaming.examples.windowing. TopSpeedWindowing
With default configuration I have no errors.

When I change the state backend with RocksDB I receive this error:

java.lang.RuntimeException: Could not forward element to next operator
                at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:388)
                at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:1)
                at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:393)
                at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:1)
                at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
                at org.apache.flink.streaming.api.functions.windowing.PassThroughWindowFunction.apply(PassThroughWindowFunction.java:32)
                at org.apache.flink.streaming.api.functions.windowing.ReduceApplyWindowFunction.apply(ReduceApplyWindowFunction.java:56)
                at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableWindowFunction.apply(InternalIterableWindowFunction.java:50)
                at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableWindowFunction.apply(InternalIterableWindowFunction.java:1)
                at org.apache.flink.streaming.runtime.operators.windowing.EvictingWindowOperator.fire(EvictingWindowOperator.java:334)
                at org.apache.flink.streaming.runtime.operators.windowing.EvictingWindowOperator.processElement(EvictingWindowOperator.java:199)
                at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:177)
                at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
                at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:270)
                at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
                at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
                at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:103)
                at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:30)
                at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:383)
                ... 15 more
Till Rohrmann
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: TopSpeedWindowing - in error: Could not forward element to next operator

Till Rohrmann
Hi Ovidiu,

which version of Flink are you using?

Cheers,
Till

On Thu, Oct 20, 2016 at 6:38 PM, Ovidiu Cristian Marcu <
[hidden email]> wrote:

> Could you check the following issue on master?
>
> When running this example org.apache.flink.streaming.examples.windowing.
> TopSpeedWindowing
> With default configuration I have no errors.
>
> When I change the state backend with RocksDB I receive this error:
>
> java.lang.RuntimeException: Could not forward element to next operator
>                 at org.apache.flink.streaming.runtime.tasks.OperatorChain$
> CopyingChainingOutput.collect(OperatorChain.java:388)
>                 at org.apache.flink.streaming.runtime.tasks.OperatorChain$
> ChainingOutput.collect(OperatorChain.java:1)
>                 at org.apache.flink.streaming.api.operators.
> AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:
> 393)
>                 at org.apache.flink.streaming.api.operators.
> AbstractStreamOperator$CountingOutput.collect(
> AbstractStreamOperator.java:1)
>                 at org.apache.flink.streaming.api.operators.
> TimestampedCollector.collect(TimestampedCollector.java:51)
>                 at org.apache.flink.streaming.api.functions.windowing.
> PassThroughWindowFunction.apply(PassThroughWindowFunction.java:32)
>                 at org.apache.flink.streaming.api.functions.windowing.
> ReduceApplyWindowFunction.apply(ReduceApplyWindowFunction.java:56)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> functions.InternalIterableWindowFunction.apply(
> InternalIterableWindowFunction.java:50)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> functions.InternalIterableWindowFunction.apply(
> InternalIterableWindowFunction.java:1)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> EvictingWindowOperator.fire(EvictingWindowOperator.java:334)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> EvictingWindowOperator.processElement(EvictingWindowOperator.java:199)
>                 at org.apache.flink.streaming.runtime.io.
> StreamInputProcessor.processInput(StreamInputProcessor.java:177)
>                 at org.apache.flink.streaming.runtime.tasks.
> OneInputStreamTask.run(OneInputStreamTask.java:66)
>                 at org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:270)
>                 at org.apache.flink.runtime.taskmanager.Task.run(Task.
> java:609)
>                 at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>                 at org.apache.flink.api.java.typeutils.runtime.
> TupleSerializer.copy(TupleSerializer.java:103)
>                 at org.apache.flink.api.java.typeutils.runtime.
> TupleSerializer.copy(TupleSerializer.java:30)
>                 at org.apache.flink.streaming.runtime.tasks.OperatorChain$
> CopyingChainingOutput.collect(OperatorChain.java:383)
>                 ... 15 more
>
Ovidiu Cristian Marcu
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: TopSpeedWindowing - in error: Could not forward element to next operator

Ovidiu Cristian Marcu
Hi

I have the latest source code Master.
I think the refactoring you were doing on State Backend gives this issue, unless the example is not supposed to work
With rocksdb.
At some point collected records's values are  null, giving the error..

What do you think?

Best,
Ovidiu

-----Original Message-----
From: Till Rohrmann [mailto:[hidden email]]
Sent: Friday, October 21, 2016 2:09 PM
To: [hidden email]
Subject: Re: TopSpeedWindowing - in error: Could not forward element to next operator

Hi Ovidiu,

which version of Flink are you using?

Cheers,
Till

On Thu, Oct 20, 2016 at 6:38 PM, Ovidiu Cristian Marcu < [hidden email]> wrote:

> Could you check the following issue on master?
>
> When running this example org.apache.flink.streaming.examples.windowing.
> TopSpeedWindowing
> With default configuration I have no errors.
>
> When I change the state backend with RocksDB I receive this error:
>
> java.lang.RuntimeException: Could not forward element to next operator
>                 at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$
> CopyingChainingOutput.collect(OperatorChain.java:388)
>                 at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$
> ChainingOutput.collect(OperatorChain.java:1)
>                 at org.apache.flink.streaming.api.operators.
> AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:
> 393)
>                 at org.apache.flink.streaming.api.operators.
> AbstractStreamOperator$CountingOutput.collect(
> AbstractStreamOperator.java:1)
>                 at org.apache.flink.streaming.api.operators.
> TimestampedCollector.collect(TimestampedCollector.java:51)
>                 at org.apache.flink.streaming.api.functions.windowing.
> PassThroughWindowFunction.apply(PassThroughWindowFunction.java:32)
>                 at org.apache.flink.streaming.api.functions.windowing.
> ReduceApplyWindowFunction.apply(ReduceApplyWindowFunction.java:56)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> functions.InternalIterableWindowFunction.apply(
> InternalIterableWindowFunction.java:50)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> functions.InternalIterableWindowFunction.apply(
> InternalIterableWindowFunction.java:1)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> EvictingWindowOperator.fire(EvictingWindowOperator.java:334)
>                 at org.apache.flink.streaming.runtime.operators.windowing.
> EvictingWindowOperator.processElement(EvictingWindowOperator.java:199)
>                 at org.apache.flink.streaming.runtime.io.
> StreamInputProcessor.processInput(StreamInputProcessor.java:177)
>                 at org.apache.flink.streaming.runtime.tasks.
> OneInputStreamTask.run(OneInputStreamTask.java:66)
>                 at org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:270)
>                 at org.apache.flink.runtime.taskmanager.Task.run(Task.
> java:609)
>                 at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>                 at org.apache.flink.api.java.typeutils.runtime.
> TupleSerializer.copy(TupleSerializer.java:103)
>                 at org.apache.flink.api.java.typeutils.runtime.
> TupleSerializer.copy(TupleSerializer.java:30)
>                 at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$
> CopyingChainingOutput.collect(OperatorChain.java:383)
>                 ... 15 more
>
Aljoscha Krettek-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: TopSpeedWindowing - in error: Could not forward element to next operator

Aljoscha Krettek-2
Hi,
the problem is that EvictingWindowOperator uses StreamRecordSerializer to
serialise the contents of the windows. This does not serialise timestamps
so when the objects are deserialised from RocksDB they all have
Long.MIN_VALUE as timestamp. The evictor in the program therefore always
evicts all elements and the window function always sees an empty iterable.

I have a fix for this in a recent PR:
https://github.com/apache/flink/pull/2656

Cheers,
Aljoscha

On Fri, 21 Oct 2016 at 16:57 Ovidiu Cristian Marcu <
[hidden email]> wrote:

> Hi
>
> I have the latest source code Master.
> I think the refactoring you were doing on State Backend gives this issue,
> unless the example is not supposed to work
> With rocksdb.
> At some point collected records's values are  null, giving the error..
>
> What do you think?
>
> Best,
> Ovidiu
>
> -----Original Message-----
> From: Till Rohrmann [mailto:[hidden email]]
> Sent: Friday, October 21, 2016 2:09 PM
> To: [hidden email]
> Subject: Re: TopSpeedWindowing - in error: Could not forward element to
> next operator
>
> Hi Ovidiu,
>
> which version of Flink are you using?
>
> Cheers,
> Till
>
> On Thu, Oct 20, 2016 at 6:38 PM, Ovidiu Cristian Marcu <
> [hidden email]> wrote:
>
> > Could you check the following issue on master?
> >
> > When running this example org.apache.flink.streaming.examples.windowing.
> > TopSpeedWindowing
> > With default configuration I have no errors.
> >
> > When I change the state backend with RocksDB I receive this error:
> >
> > java.lang.RuntimeException: Could not forward element to next operator
> >                 at
> > org.apache.flink.streaming.runtime.tasks.OperatorChain$
> > CopyingChainingOutput.collect(OperatorChain.java:388)
> >                 at
> > org.apache.flink.streaming.runtime.tasks.OperatorChain$
> > ChainingOutput.collect(OperatorChain.java:1)
> >                 at org.apache.flink.streaming.api.operators.
> >
> AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:
> > 393)
> >                 at org.apache.flink.streaming.api.operators.
> > AbstractStreamOperator$CountingOutput.collect(
> > AbstractStreamOperator.java:1)
> >                 at org.apache.flink.streaming.api.operators.
> > TimestampedCollector.collect(TimestampedCollector.java:51)
> >                 at org.apache.flink.streaming.api.functions.windowing.
> > PassThroughWindowFunction.apply(PassThroughWindowFunction.java:32)
> >                 at org.apache.flink.streaming.api.functions.windowing.
> > ReduceApplyWindowFunction.apply(ReduceApplyWindowFunction.java:56)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > functions.InternalIterableWindowFunction.apply(
> > InternalIterableWindowFunction.java:50)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > functions.InternalIterableWindowFunction.apply(
> > InternalIterableWindowFunction.java:1)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > EvictingWindowOperator.fire(EvictingWindowOperator.java:334)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > EvictingWindowOperator.processElement(EvictingWindowOperator.java:199)
> >                 at org.apache.flink.streaming.runtime.io.
> > StreamInputProcessor.processInput(StreamInputProcessor.java:177)
> >                 at org.apache.flink.streaming.runtime.tasks.
> > OneInputStreamTask.run(OneInputStreamTask.java:66)
> >                 at org.apache.flink.streaming.runtime.tasks.StreamTask.
> > invoke(StreamTask.java:270)
> >                 at org.apache.flink.runtime.taskmanager.Task.run(Task.
> > java:609)
> >                 at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NullPointerException
> >                 at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.copy(TupleSerializer.java:103)
> >                 at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.copy(TupleSerializer.java:30)
> >                 at
> > org.apache.flink.streaming.runtime.tasks.OperatorChain$
> > CopyingChainingOutput.collect(OperatorChain.java:383)
> >                 ... 15 more
> >
>
Ovidiu Cristian Marcu
Reply | Threaded
Open this post in threaded view
♦
♦
|

RE: TopSpeedWindowing - in error: Could not forward element to next operator

Ovidiu Cristian Marcu
Thank you, I will check this fix in my environment.

Best,
Ovidiu

-----Original Message-----
From: Aljoscha Krettek [mailto:[hidden email]]
Sent: Friday, October 21, 2016 5:47 PM
To: [hidden email]
Subject: Re: TopSpeedWindowing - in error: Could not forward element to next operator

Hi,
the problem is that EvictingWindowOperator uses StreamRecordSerializer to serialise the contents of the windows. This does not serialise timestamps so when the objects are deserialised from RocksDB they all have Long.MIN_VALUE as timestamp. The evictor in the program therefore always evicts all elements and the window function always sees an empty iterable.

I have a fix for this in a recent PR:
https://github.com/apache/flink/pull/2656

Cheers,
Aljoscha

On Fri, 21 Oct 2016 at 16:57 Ovidiu Cristian Marcu < [hidden email]> wrote:

> Hi
>
> I have the latest source code Master.
> I think the refactoring you were doing on State Backend gives this
> issue, unless the example is not supposed to work With rocksdb.
> At some point collected records's values are  null, giving the error..
>
> What do you think?
>
> Best,
> Ovidiu
>
> -----Original Message-----
> From: Till Rohrmann [mailto:[hidden email]]
> Sent: Friday, October 21, 2016 2:09 PM
> To: [hidden email]
> Subject: Re: TopSpeedWindowing - in error: Could not forward element
> to next operator
>
> Hi Ovidiu,
>
> which version of Flink are you using?
>
> Cheers,
> Till
>
> On Thu, Oct 20, 2016 at 6:38 PM, Ovidiu Cristian Marcu <
> [hidden email]> wrote:
>
> > Could you check the following issue on master?
> >
> > When running this example org.apache.flink.streaming.examples.windowing.
> > TopSpeedWindowing
> > With default configuration I have no errors.
> >
> > When I change the state backend with RocksDB I receive this error:
> >
> > java.lang.RuntimeException: Could not forward element to next operator
> >                 at
> > org.apache.flink.streaming.runtime.tasks.OperatorChain$
> > CopyingChainingOutput.collect(OperatorChain.java:388)
> >                 at
> > org.apache.flink.streaming.runtime.tasks.OperatorChain$
> > ChainingOutput.collect(OperatorChain.java:1)
> >                 at org.apache.flink.streaming.api.operators.
> >
> AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:
> > 393)
> >                 at org.apache.flink.streaming.api.operators.
> > AbstractStreamOperator$CountingOutput.collect(
> > AbstractStreamOperator.java:1)
> >                 at org.apache.flink.streaming.api.operators.
> > TimestampedCollector.collect(TimestampedCollector.java:51)
> >                 at org.apache.flink.streaming.api.functions.windowing.
> > PassThroughWindowFunction.apply(PassThroughWindowFunction.java:32)
> >                 at org.apache.flink.streaming.api.functions.windowing.
> > ReduceApplyWindowFunction.apply(ReduceApplyWindowFunction.java:56)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > functions.InternalIterableWindowFunction.apply(
> > InternalIterableWindowFunction.java:50)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > functions.InternalIterableWindowFunction.apply(
> > InternalIterableWindowFunction.java:1)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > EvictingWindowOperator.fire(EvictingWindowOperator.java:334)
> >                 at
> org.apache.flink.streaming.runtime.operators.windowing.
> > EvictingWindowOperator.processElement(EvictingWindowOperator.java:199)
> >                 at org.apache.flink.streaming.runtime.io.
> > StreamInputProcessor.processInput(StreamInputProcessor.java:177)
> >                 at org.apache.flink.streaming.runtime.tasks.
> > OneInputStreamTask.run(OneInputStreamTask.java:66)
> >                 at org.apache.flink.streaming.runtime.tasks.StreamTask.
> > invoke(StreamTask.java:270)
> >                 at org.apache.flink.runtime.taskmanager.Task.run(Task.
> > java:609)
> >                 at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NullPointerException
> >                 at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.copy(TupleSerializer.java:103)
> >                 at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.copy(TupleSerializer.java:30)
> >                 at
> > org.apache.flink.streaming.runtime.tasks.OperatorChain$
> > CopyingChainingOutput.collect(OperatorChain.java:383)
> >                 ... 15 more
> >
>
Free forum by Nabble Edit this page