Flink Parallelism for various type of transformation

[Prasanna kumar]

Hi ,

I used t2.medium machines for the task manager nodes. It has 2 CPU and 4GB memory.

But the task manager screen shows that there are 4 slots. 

Generally we should match the number of slots to the number of cores. 

Our pipeline is Source -> Simple Transform -> Sink.

What happens when we have more slots than cores in following scenarios?

1) The transform is just changing of json format.

2)  When the transformation is done by hitting another server (HTTP request) 

Thanks,

Prasanna.

[Arvid Heise-3]

Hi Prasanna,

overcommitting cores was actually a recommended technique a while ago to counter-balance I/O. So it's not bad per se.

However, with slot sharing each core is already doing the work for source, transform, sink, so it's not necessary. So I'd go with slots = cores and I rather strongly suggest to switch to async I/O to perform the external transformation. [1]

[1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/asyncio.html

On Mon, Jul 6, 2020 at 7:01 PM Prasanna kumar <[hidden email]> wrote:

Hi ,

I used t2.medium machines for the task manager nodes. It has 2 CPU and 4GB memory.

But the task manager screen shows that there are 4 slots. 

Generally we should match the number of slots to the number of cores. 

Our pipeline is Source -> Simple Transform -> Sink.

What happens when we have more slots than cores in following scenarios?

1) The transform is just changing of json format.

2)  When the transformation is done by hitting another server (HTTP request) 

Thanks,

Prasanna.

--