TaskManager Transfer buffers

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

TaskManager Transfer buffers

David Herzog
Hi dear Support,

I am currently working on a research project and have to understand how the
transfer of buffers exactly works.
https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
This page describes how a request for a ResultSubpartion works.
I got to the part that
*PartitionRequestClient.requestSubpartition()*
actually sends the request to the other task, but I couldn't figure out how
and where this request is received and processed in the receiving task.
I would be very happ if you could help me trying to understand where
requests for ResultSubpartitions from other Tasks are handled and how the
ResultPartition actually answers those request.
Thank you in advance :)

Regards
David Herzog
Reply | Threaded
Open this post in threaded view
|

Re: TaskManager Transfer buffers

Ufuk Celebi-2
Hey David! Yes, sure.

- The NettyServer handles these requests in the server pipeline (see
PartitionRequestProtocol#getServerChannelHandlers())
- The main handler for these requests is
PartitionRequestServerHandler: it gets the requests, forwards it to
the ResultPartitionProvider and queues it for transfer.
- Data transfer happens via PartitionRequestQueue

Are these pointers enough? If you need further input, feel free to ping me.

– Ufuk

On Tue, Aug 9, 2016 at 6:41 PM, David Herzog <[hidden email]> wrote:

> Hi dear Support,
>
> I am currently working on a research project and have to understand how the
> transfer of buffers exactly works.
> https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
> This page describes how a request for a ResultSubpartion works.
> I got to the part that
> *PartitionRequestClient.requestSubpartition()*
> actually sends the request to the other task, but I couldn't figure out how
> and where this request is received and processed in the receiving task.
> I would be very happ if you could help me trying to understand where
> requests for ResultSubpartitions from other Tasks are handled and how the
> ResultPartition actually answers those request.
> Thank you in advance :)
>
> Regards
> David Herzog