Kafka source stuck while canceling

[Gyula Fóra]

Hey guys,

Has anyone ever got something similar working with the kafka sources?

11:52:48,838 WARN  org.apache.flink.runtime.taskmanager.Task
      - Task 'Source: Kafka[***] (3/4)' did not react to cancelling signal,
but is stuck in method:
 org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:208)
org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
java.lang.Thread.run(Thread.java:745)

The failure was caused by a different operator in the pipeline, but the job
could never be fully cancelled and restarted due to this error.

Any idea is appreciated :)

Cheers,
Gyula

[Stephan Ewen]

The Kafka consumer API has issue with being interruptible that will
hopefully get better in Kafka 0.8.3.

There must be another issue here in addition, though. Does the stacktrace
go any deeper than that?
I would assume that the main invokable thread is stuck in some blocking
method, or in a loop that does not terminate. It might also be stuck on a
lock, in which case it would be waiting for the lock holder to terminate.

Do you have the traces from other threads as well, so we could look which
one actually is stuck while holding the lock?

Greetings,
Stephan


On Mon, Oct 19, 2015 at 12:06 PM, Gyula Fóra <[hidden email]> wrote:

> Hey guys,
>
> Has anyone ever got something similar working with the kafka sources?
>
> 11:52:48,838 WARN  org.apache.flink.runtime.taskmanager.Task
>       - Task 'Source: Kafka[***] (3/4)' did not react to cancelling signal,
> but is stuck in method:
>
>  org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:208)
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> java.lang.Thread.run(Thread.java:745)
>
> The failure was caused by a different operator in the pipeline, but the job
> could never be fully cancelled and restarted due to this error.
>
> Any idea is appreciated :)
>
> Cheers,
> Gyula
>

... [show rest of quote]

[Gyula Fóra]

I don't think I have the logs for that run anymore, I wasn't smart enough
to save it.

If I encounter it again I will try to get more info.

Thanks,
Gyula

Stephan Ewen <[hidden email]> ezt írta (időpont: 2015. okt. 21., Sze,
19:04):

> The Kafka consumer API has issue with being interruptible that will
> hopefully get better in Kafka 0.8.3.
>
> There must be another issue here in addition, though. Does the stacktrace
> go any deeper than that?
> I would assume that the main invokable thread is stuck in some blocking
> method, or in a loop that does not terminate. It might also be stuck on a
> lock, in which case it would be waiting for the lock holder to terminate.
>
> Do you have the traces from other threads as well, so we could look which
> one actually is stuck while holding the lock?
>
> Greetings,
> Stephan
>
>
> On Mon, Oct 19, 2015 at 12:06 PM, Gyula Fóra <[hidden email]> wrote:
>
> > Hey guys,
> >
> > Has anyone ever got something similar working with the kafka sources?
> >
> > 11:52:48,838 WARN  org.apache.flink.runtime.taskmanager.Task
> >       - Task 'Source: Kafka[***] (3/4)' did not react to cancelling
> signal,
> > but is stuck in method:
> >
> >
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:208)
> > org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> > java.lang.Thread.run(Thread.java:745)
> >
> > The failure was caused by a different operator in the pipeline, but the
> job
> > could never be fully cancelled and restarted due to this error.
> >
> > Any idea is appreciated :)
> >
> > Cheers,
> > Gyula
> >
>

... [show rest of quote]