Too many open files

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Too many open files

Govindarajan Srinivasaraghavan
Hi,

We have a streaming job that runs on flink in docker and checkpointing
happens every 10 seconds. After several starts and cancellations we are
facing this issue with file handles.

The job reads data from kafka, processes it and writes it back to kafka and
we are using RocksDB state backend. For now we have increased the number
file handles to resolve the problem but would like to know if this is
expected or is it an issue. Thanks.

java.io.FileNotFoundException:
/tmp/flink-io-b3043cd6-50c8-446a-8c25-fade1b1862c0/cb317fc2578db72b3046468948fa00f2f17039b6104e72fb8c58938e5869cfbc.0.buffer
(Too many open files)

        at java.io.RandomAccessFile.open0(Native Method)

        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)

        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)

        at
org.apache.flink.streaming.runtime.io.BufferSpiller.createSpillingChannel(BufferSpiller.java:259)

        at
org.apache.flink.streaming.runtime.io.BufferSpiller.<init>(BufferSpiller.java:120)

        at
org.apache.flink.streaming.runtime.io.BarrierBuffer.<init>(BarrierBuffer.java:149)

        at
org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.<init>(StreamTwoInputProcessor.java:147)

        at
org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.init(TwoInputStreamTask.java:79)

        at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:235)

        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)

        at java.lang.Thread.run(Thread.java:748)

Regards,
Govind
Reply | Threaded
Open this post in threaded view
|

Re: Too many open files

Ted Yu
Not sure if you have looked at FLINK-8707

FYI

On Tue, Mar 20, 2018 at 2:13 PM, Govindarajan Srinivasaraghavan <
[hidden email]> wrote:

> Hi,
>
> We have a streaming job that runs on flink in docker and checkpointing
> happens every 10 seconds. After several starts and cancellations we are
> facing this issue with file handles.
>
> The job reads data from kafka, processes it and writes it back to kafka and
> we are using RocksDB state backend. For now we have increased the number
> file handles to resolve the problem but would like to know if this is
> expected or is it an issue. Thanks.
>
> java.io.FileNotFoundException:
> /tmp/flink-io-b3043cd6-50c8-446a-8c25-fade1b1862c0/
> cb317fc2578db72b3046468948fa00f2f17039b6104e72fb8c58938e5869cfbc.0.buffer
> (Too many open files)
>
>         at java.io.RandomAccessFile.open0(Native Method)
>
>         at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
>
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
>
>         at
> org.apache.flink.streaming.runtime.io.BufferSpiller.createSpillingChannel(
> BufferSpiller.java:259)
>
>         at
> org.apache.flink.streaming.runtime.io.BufferSpiller.<
> init>(BufferSpiller.java:120)
>
>         at
> org.apache.flink.streaming.runtime.io.BarrierBuffer.<
> init>(BarrierBuffer.java:149)
>
>         at
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.<init>
> (StreamTwoInputProcessor.java:147)
>
>         at
> org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.init(
> TwoInputStreamTask.java:79)
>
>         at
> org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:235)
>
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
>
>         at java.lang.Thread.run(Thread.java:748)
>
> Regards,
> Govind
>