Yingjie Cao created FLINK-15455:
-----------------------------------
Summary: Enable TCP connection reuse across multiple jobs.
Key: FLINK-15455
URL:
https://issues.apache.org/jira/browse/FLINK-15455 Project: Flink
Issue Type: Improvement
Components: Runtime / Network
Reporter: Yingjie Cao
Currently, tcp connections can be only reuse by tasks residing in the same TaskManager and consumes the same IntermediateResult. And after job finish or failover, the TCP connections are closed and new connections must be setup latter.
As an improvement, we can make tcp connections a cluster level resource which can be reused by multi jobs. The advantages are as follows:
# Reduce the number of TCP connections so we can save some resources.
# Reduce the overhead of connection setup and close so restarted jobs after failover and latter jobs submitted to the same session cluster can reuse the previous connections.
We use Flink session cluster as a service for ad-hoc queries and the users can produce some statistics or create some statements and reports at any time. Most of the queries finish in 2s and we find tcp connection reuse help a lot to reduce the average execution time which means more queries can be processed using the same resource and time with even better user experience.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
