Till Rohrmann created FLINK-17947:
-------------------------------------
Summary: Retry REST requests if RpcEndpoint died before responding to request
Key: FLINK-17947
URL:
https://issues.apache.org/jira/browse/FLINK-17947 Project: Flink
Issue Type: Improvement
Components: Runtime / REST
Affects Versions: 1.10.1, 1.11.0
Reporter: Till Rohrmann
Fix For: 1.12.0
Currently, it can happen that a REST handler sends a request to a leader {{RpcEndpoint}} and before the {{RpcEndpoint}} has a chance to respond, it might shut down (e.g. due to losing the leadership). In this case, the {{ActorSystem}} will send an {{AskTimeoutException}} as the response with the message {{Recipient Actor[akka://flink/user/rpc/dispatcher_1#-1875884516] had already been terminated.}}. This exception will be treated as any other exception and forwarded to the REST client. There it will be treated as a normal timeout exception which causes the operation (e.g. requesting job details) to fail.
I was wondering whether this case should not be handled slightly differently. If the REST handler would respond with a {{SERVICE_UNAVAILABLE}}} HTTP response code, then the {{RestClusterClient}} would retry the operation. One could think of it as if there wouldn't have been a leader available before. This is similar to the situation when there is no current leader and we are waiting for the leader election to finish. Alternatively, we could extend the {{RestClusterClient.isConnectionProblemOrServiceUnavailable}} predicate to also cover the case of special {{AskTimeoutExceptions}}.
cc [~chesnay]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
