Fritz Budiyanto created FLINK-17853:
---------------------------------------
Summary: JobGraph is not getting deleted after Job cancelation
Key: FLINK-17853
URL:
https://issues.apache.org/jira/browse/FLINK-17853 Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.9.2
Environment: Flink 1.9.2
Zookeeper from AWS MSK
Reporter: Fritz Budiyanto
Attachments: flinkissue.txt
I have been seeing this issue several time where JobGraph are not cleaned up properly after Job deletion. Job deletion is performed by using "flink stop" command. As a result JobGraph node lingering in ZK, when Flink cluster is restarted, it will attempt to do HA restoration on non existing checkpoint which prevent the Flink cluster to come up.
2020-05-19 19:56:21,471 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task Source: kafkaConsumer[update_server] -> (DetectedUpdateMessageConverter -> Sink: update_server.detected_updates, DrivenCoordinatesMessageConverter -> Sink: update_server.driven_coordinates) 588902a8096f49845b09fa1f595d6065.
2020-05-19 19:56:21,622 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Free slot TaskSlot(index:0, state:ACTIVE, resource profile: ResourceProfile\{cpuCores=1.7976931348623157E308, heapMemoryInMB=2147483647, directMemoryInMB=2147483647, nativeMemoryInMB=2147483647, networkMemoryInMB=2147483647, managedMemoryInMB=642}, allocationId: 29f6a5f83c832486f2d7ebe5c779fa32, jobId: 86a028b3f7aada8ffe59859ca71d6385).
2020-05-19 19:56:21,622 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Remove job 86a028b3f7aada8ffe59859ca71d6385 from job leader monitoring.
2020-05-19 19:56:21,622 INFO org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService - Stopping ZooKeeperLeaderRetrievalService /leader/86a028b3f7aada8ffe59859ca71d6385/job_manager_lock.
2020-05-19 19:56:21,623 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close JobManager connection for job 86a028b3f7aada8ffe59859ca71d6385.
2020-05-19 19:56:21,624 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close JobManager connection for job 86a028b3f7aada8ffe59859ca71d6385.
2020-05-19 19:56:21,624 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Cannot reconnect to job 86a028b3f7aada8ffe59859ca71d6385 because it is not registered.
...
Zookeeper CLI:
ls /flink/cluster_update/jobgraphs
[86a028b3f7aada8ffe59859ca71d6385]
Attached is the Flink logs in reverse order
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
