How to modify TestBaseUtils.TASK_MANAGER_MEMORY_SIZE

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to modify TestBaseUtils.TASK_MANAGER_MEMORY_SIZE

André Petermann
Hi all,

in longer workflows one reaches "too few memory segments error" in Tests
inheriting from TestBaseUtils. A solution to this problem is increasing
TASK_MANAGER_MEMORY_SIZE in the cluster configuration.

We are doing this is a custom @BeforeClass method replacing the logic of
the TestBaseUtils.startCluster(...) methods. At that, we do not only set
TASK_MANAGER_MEMORY_SIZE but all other configurations, too. However,
we'd prefer only setting a single value and reuse the original methods.
Is there a way to do this?

If not, how do you think about an issue about an external configuration
option for TestBaseUtils.TASK_MANAGER_MEMORY_SIZE?

Cheers,
Andre

--
-------------------------------------------
PhD Student
University of Leipzig
Department of Computer Science
Database Research Group

email: [hidden email]
web:   dbs.uni-leipzig.de
-------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: How to modify TestBaseUtils.TASK_MANAGER_MEMORY_SIZE

Till Rohrmann
Hi Andre,

what you could do is to either add a new startCluster method which takes as
a parameter the task manager memory size. Alternatively, and I would prefer
this alternative, you could change the startCluster(Configuration,
StreamingMode, boolean) method to always overwrite the configuration values
if they are set. Thus, first test if, for example, the
ConfigConstants.TASK_MANAGER_MEMORY_SIZE_KEY is set and only iff it’s not
set, then set it to TestBaseUtils.TASK_MANAGER_MEMORY_SIZE.

That way, you have more control over your cluster and don’t run into a
combinatorial explosion if you want to change other options sometime later.

Cheers,
Till


On Fri, Nov 20, 2015 at 4:11 PM, André Petermann <
[hidden email]> wrote:

> Hi all,
>
> in longer workflows one reaches "too few memory segments error" in Tests
> inheriting from TestBaseUtils. A solution to this problem is increasing
> TASK_MANAGER_MEMORY_SIZE in the cluster configuration.
>
> We are doing this is a custom @BeforeClass method replacing the logic of
> the TestBaseUtils.startCluster(...) methods. At that, we do not only set
> TASK_MANAGER_MEMORY_SIZE but all other configurations, too. However, we'd
> prefer only setting a single value and reuse the original methods. Is there
> a way to do this?
>
> If not, how do you think about an issue about an external configuration
> option for TestBaseUtils.TASK_MANAGER_MEMORY_SIZE?
>
> Cheers,
> Andre
>
> --
> -------------------------------------------
> PhD Student
> University of Leipzig
> Department of Computer Science
> Database Research Group
>
> email: [hidden email]
> web:   dbs.uni-leipzig.de
> -------------------------------------------
>