Hi Andrey,
Thanks for concluding this valuable proposal. It really extends and enhances the details of partition lifecycle management ever mentioned in FLIP-31. I reviewed the google doc and agreed the general points in it.
One implict point in doc is that the shuffle is not limted to only serve for downstream and upstream in one job, even the data shuffle is required across jobs like interactive feature, so the ShuffleMaster is not coupled with JobMaster.
These thoughts are also consitent with our previous confirmation and we could fight in this direction in long term. We would further focus on them in specific PRs considering the feasibility in next 1.9 release.
Best,
Zhijiang
------------------------------------------------------------------
From:Andrey Zagrebin <
[hidden email]>
Send Time:2019年3月29日(星期五) 18:38
To:dev <
[hidden email]>
Subject:[Proposal] Shuffle resources lifecycle management
Hi All,
While working on pluggable shuffle architecture, looking into interactive
programming and fine-grained recovery efforts, we released that lifecycle
management of intermediate result partitions needs more detailed
discussion to enable envisioned use cases.
Here I prepared a design document to address this concern. The document
proposes certain extensions to FLIP-31 (Design of Pluggable Shuffle
Service):
https://docs.google.com/document/d/13vAJJxfRXAwI4MtO8dux8hHnNMw2Biu5XRrb_hvGehALooking forward to your feedback.
Thanks,
Andrey
Locked
