Memory segment error when migrating functional code from Flink 0.9 to 0.8

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory segment error when migrating functional code from Flink 0.9 to 0.8

Andra Lungu
Hello everyone,

I am implementing a graph algorithm as part of a course and I will also add
it to the Flink- Gelly examples.
My problem is that I started developing it in the Gelly repository, which
runs on flink 0.9. It works like a charm there, but in order to test in on
a cluster to see its real capabilities, I need to move it to the course
repository, which runs on flink 0.8.

Initially, I thought this migration should occur without incidents since
flink 0.8 is more stable. Instead, I got the following exception:
java.lang.IllegalArgumentException: *Too few memory segments provided. Hash
Table needs at least 33 memory segments.*
    at
org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:238)
    at
org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:227)
    at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177)
    at
org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279)
    at
org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
    at
org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
    at java.lang.Thread.run(Thread.java:745)

This is the code for Gelly, where all tests pass:
https://github.com/andralungu/flink-graph/tree/minSpanningTree
Unfortunately, the code for the course is private, so you cannot actually
see it... maybe @aalexandrov can do something about the privacy settings of
this repo.
https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm

Thank you!
Andra
Reply | Threaded
Open this post in threaded view
|

Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8

Till Rohrmann
Hi Andra,

have you tried increasing the number of network buffers in your cluster?
You can control by the configuration value:

taskmanager.network.numberOfBuffers: #numberBuffers

Greets,

Till

On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu <[hidden email]> wrote:

> Hello everyone,
>
> I am implementing a graph algorithm as part of a course and I will also add
> it to the Flink- Gelly examples.
> My problem is that I started developing it in the Gelly repository, which
> runs on flink 0.9. It works like a charm there, but in order to test in on
> a cluster to see its real capabilities, I need to move it to the course
> repository, which runs on flink 0.8.
>
> Initially, I thought this migration should occur without incidents since
> flink 0.8 is more stable. Instead, I got the following exception:
> java.lang.IllegalArgumentException: *Too few memory segments provided. Hash
> Table needs at least 33 memory segments.*
>     at
>
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:238)
>     at
>
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:227)
>     at
>
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177)
>     at
>
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279)
>     at
>
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>     at
>
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
>     at java.lang.Thread.run(Thread.java:745)
>
> This is the code for Gelly, where all tests pass:
> https://github.com/andralungu/flink-graph/tree/minSpanningTree
> Unfortunately, the code for the course is private, so you cannot actually
> see it... maybe @aalexandrov can do something about the privacy settings of
> this repo.
> https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm
>
> Thank you!
> Andra
>
Reply | Threaded
Open this post in threaded view
|

Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8

Stephan Ewen
This is actually a problem of the number of memory segments available to
the hash table for the solution set.

For complex pipelines, memory currently gets too fragmented.

There are two workarounds, until we do the dynamic memory management, or
break it into shorter pipelines: Break the job up into subparts, or move
the solution set to the user code part of the heap. There is a flag to
attach to the Delta Iterations, see
"DeltaIteration#setSolutionSetUnManaged()"

Greetinigs,
Stephan


On Mon, Feb 9, 2015 at 10:32 AM, Till Rohrmann <[hidden email]> wrote:

> Hi Andra,
>
> have you tried increasing the number of network buffers in your cluster?
> You can control by the configuration value:
>
> taskmanager.network.numberOfBuffers: #numberBuffers
>
> Greets,
>
> Till
>
> On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu <[hidden email]> wrote:
>
> > Hello everyone,
> >
> > I am implementing a graph algorithm as part of a course and I will also
> add
> > it to the Flink- Gelly examples.
> > My problem is that I started developing it in the Gelly repository, which
> > runs on flink 0.9. It works like a charm there, but in order to test in
> on
> > a cluster to see its real capabilities, I need to move it to the course
> > repository, which runs on flink 0.8.
> >
> > Initially, I thought this migration should occur without incidents since
> > flink 0.8 is more stable. Instead, I got the following exception:
> > java.lang.IllegalArgumentException: *Too few memory segments provided.
> Hash
> > Table needs at least 33 memory segments.*
> >     at
> >
> >
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:238)
> >     at
> >
> >
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:227)
> >     at
> >
> >
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177)
> >     at
> >
> >
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279)
> >     at
> >
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
> >     at
> >
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
> >     at java.lang.Thread.run(Thread.java:745)
> >
> > This is the code for Gelly, where all tests pass:
> > https://github.com/andralungu/flink-graph/tree/minSpanningTree
> > Unfortunately, the code for the course is private, so you cannot actually
> > see it... maybe @aalexandrov can do something about the privacy settings
> of
> > this repo.
> > https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm
> >
> > Thank you!
> > Andra
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8

Andra Lungu
Hi Stephan,

I tried the solution with DeltaIteration#setSolutionSetUnManaged(),
unfortunatelly the error is still there... Even when I try to run it with
just one iteration... Also, I am not sure that the job can be broken into
subparts in this particular case.

Any other suggestions would be appreciated :)
Thanks!
Andra

On Mon, Feb 9, 2015 at 10:36 AM, Stephan Ewen <[hidden email]> wrote:

> This is actually a problem of the number of memory segments available to
> the hash table for the solution set.
>
> For complex pipelines, memory currently gets too fragmented.
>
> There are two workarounds, until we do the dynamic memory management, or
> break it into shorter pipelines: Break the job up into subparts, or move
> the solution set to the user code part of the heap. There is a flag to
> attach to the Delta Iterations, see
> "DeltaIteration#setSolutionSetUnManaged()"
>
> Greetinigs,
> Stephan
>
>
> On Mon, Feb 9, 2015 at 10:32 AM, Till Rohrmann <[hidden email]>
> wrote:
>
> > Hi Andra,
> >
> > have you tried increasing the number of network buffers in your cluster?
> > You can control by the configuration value:
> >
> > taskmanager.network.numberOfBuffers: #numberBuffers
> >
> > Greets,
> >
> > Till
> >
> > On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu <[hidden email]>
> wrote:
> >
> > > Hello everyone,
> > >
> > > I am implementing a graph algorithm as part of a course and I will also
> > add
> > > it to the Flink- Gelly examples.
> > > My problem is that I started developing it in the Gelly repository,
> which
> > > runs on flink 0.9. It works like a charm there, but in order to test in
> > on
> > > a cluster to see its real capabilities, I need to move it to the course
> > > repository, which runs on flink 0.8.
> > >
> > > Initially, I thought this migration should occur without incidents
> since
> > > flink 0.8 is more stable. Instead, I got the following exception:
> > > java.lang.IllegalArgumentException: *Too few memory segments provided.
> > Hash
> > > Table needs at least 33 memory segments.*
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:238)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:227)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
> > >     at java.lang.Thread.run(Thread.java:745)
> > >
> > > This is the code for Gelly, where all tests pass:
> > > https://github.com/andralungu/flink-graph/tree/minSpanningTree
> > > Unfortunately, the code for the course is private, so you cannot
> actually
> > > see it... maybe @aalexandrov can do something about the privacy
> settings
> > of
> > > this repo.
> > > https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm
> > >
> > > Thank you!
> > > Andra
> > >
> >
>