Till Rohrmann created FLINK-22636:
-------------------------------------
Summary: Group job specific ZooKeeper HA services under common jobs/<JobID> zNode
Key: FLINK-22636
URL:
https://issues.apache.org/jira/browse/FLINK-22636 Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Affects Versions: 1.12.3, 1.13.0, 1.14.0
Reporter: Till Rohrmann
Fix For: 1.14.0
In order to better clean up Zookeeper HA services, I suggest grouping job-specific services under a common {{jobs/<JobID>}} zNode. That way, it becomes trivial to clean up the job-specific Zookeeper data (simply deleting the {{jobs/<JobID>}} node.
Currently, our Zookeeper structure is not really structured well. The current layout looks like this:
{code}
clusterID -> jobgraphs -> <job-id>
-> checkpoints -> <job-id> -> checkpoint-1
-> checkpoint-counter -> <job-id> -> counter
-> leaderlatch -> dispatcher_lock
-> resourc_emanager_lock
-> <job-id>
-> leader -> dispatcher_lock
-> resource_manager_lock
-> <job-id>
{code}
The new layout could look like this:
{code}
clusterID -> jobgraphs -> <job-id>
-> jobs -> <job-id> -> checkpoints -> checkpoint-1
-> checkpoint_id_counter -> counter
-> leader -> latch
-> connection_info
-> leader -> dispatcher -> latch
-> connection_info
-> resource_manager -> latch
-> connection_info
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
