Till Rohrmann created FLINK-18293:
-------------------------------------
Summary: TaskExecutor offering non empty slots can lead to resource violation
Key: FLINK-18293
URL:
https://issues.apache.org/jira/browse/FLINK-18293 Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.10.1, 1.11.0
Reporter: Till Rohrmann
Fix For: 1.12.0
When a {{JobMaster}} loses leadership, then the {{TaskExecutor}} will fail all running tasks belonging to this job and transition all slots belonging to this job from {{ACTIVE}} into {{ALLOCATED}}. The idea is that these slots can be re-offered to the new leader of the very same job.
A problem arises when the {{Task}} cancellation takes longer than the election of the new leader. In this case, the slot containing a {{CANCELLING}} task, will be offered to the new {{JobMaster}} as empty. The {{JobMaster}} not knowing that the slot still contains a resource consumer might deploy new tasks into it believing that these tasks can use all of the available resources. In the best case, the newly deployed {{Tasks}} will simply get fewer resources than thought. In the worst case this will lead to a resource violation.
W/o the {{JobMaster}} being able to reconcile the state of already deployed {{Tasks}} into {{Slots}}, I believe that we should only re-offer the slot when it is free. One might model this scenario with introducing a new {{TaskSlotState.CLEANING}}. {{CLEANING}} means that the slot is still allocated for a given job but that there are still some resources which need to be cleaned up before it can be re-offered (transition to state {{ALLOCATED}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
