Xuannan Su created FLINK-19761:
----------------------------------
Summary: Add lookup method for registered ShuffleDescriptor in ShuffleMaster
Key: FLINK-19761
URL:
https://issues.apache.org/jira/browse/FLINK-19761 Project: Flink
Issue Type: Improvement
Components: Runtime / Network
Reporter: Xuannan Su
Currently, the ShuffleMaster can register a partition and get the shuffle descriptor. However, it lacks the ability to look up the registered ShuffleDescriptors belongs to an IntermediateResult by the IntermediateDataSetID.
Adding the lookup method to the ShuffleMaster can make reusing the cluster partition more easily. For example, we don't have to return the ShuffleDescriptor to the client just so that the other job can somehow encode the ShuffleDescriptor in the JobGraph to consume the cluster partition. Instead, we only need to return the IntermediateDatSetID and use it to lookup the ShuffleDescriptor by another job.
By adding the lookup method in ShuffleMaster, if we have an external shuffle service and the lifecycle of the IntermediateResult is not bounded to the cluster, we can look up the ShuffleDescriptor and reuse the IntermediateResult by a job running on another cluster even if the cluster that produced the IntermediateResult is shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
