Yun Gao created FLINK-12865:
-------------------------------
Summary: State inconsistency between RM and TM on the slot status
Key: FLINK-12865
URL:
https://issues.apache.org/jira/browse/FLINK-12865 Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Reporter: Yun Gao
Assignee: Yun Gao
There may be state inconsistency between TM and RM due to race condition and message loss:
# When TM sends heartbeat, it retrieve SlotReport in the main thread, but sends the heartbeat in another thread. There may be cases that the slot on TM is FREE initially and SlotReport read the FREE state, then RM requests slot and mark the slot as allocated, and the SlotReport finally override the allocated status at the RM side wrongly.
# When RM requests slot, TM received the requests but the acknowledge message get lot. Then RM will think this slot is free and assigned it to other request.
Add a version to the slot may solves this problem.
# Add a SYNCING status in the RM side. SYNCING means request is sent to TM but not knowing the result of the request. A slot with SYNCING status cannot be allocated to others.
# RM and TM maintains a version for each slot, and the version starts from 0.
# Whenever RM requests slot, it add the version by 1, and send the requests to TM. TM will only do allocation when RM's version > TM's version.
# TM will also attach the version in the HeartBeat and RM will only accept the slot status when the TM's version >= RM's version.
The version method is a simplified solution of the full vector clock based state management. In the full vector-clock design, the version should be a vector represents (RM's version, TM's version). Whenever RM modify the slot's status (requestSlot) and TM modify the slot's status (freeSlot), It need to first increase the corresponding component and send the sync messages, and the messages can only be accepted when the vector version >= messages' vector version.
However, since for the status of slot TM will only modify its status when freeing slots, we can ignore the component of TM's side will only cause a freed slot be marked as allocated, this will not cause error, and the free status can be finally updated to RM with Heartbeat message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
