Ufuk Celebi created FLINK-1093:
----------------------------------
Summary: Add non-multiplexing NetworkConnectionManager
Key: FLINK-1093
URL:
https://issues.apache.org/jira/browse/FLINK-1093 Project: Flink
Issue Type: Improvement
Components: Distributed Runtime
Affects Versions: 0.7-incubating
Reporter: Ufuk Celebi
Priority: Minor
When translating a JobGraph to an ExecutionGraph, edges are assigned connection indexes to ensure deadlock freedom of the network stack. Deadlocks can easily occur when multiplexing multiple logical channels over a single TCP connection, because of the limited network buffers. Each incoming envelope, which has an attached buffer, needs to request a network buffer from the receiver's input buffer pool. If the receiver does not have a buffer available, it unsubscribes from the TCP connection until a new buffer is available. This unsubscribe then affects all other logical channels, which are multiplexed over the same TCP connection. With certain user programs this can result in a deadlock.
One of the problems when running in multiplexed mode is the question of when to close TCP connections in a reliable way without loosing or re-ordering network envelopes (as evident in FLINK-1063). In non-multiplexed mode this is trivial, because every logical channel is mapped to a phyiscal channel and the close of the logical channel translates to a close of the physical channel.
However, this approach results in limited scalability as high degrees of parallelism result in a high number of TCP connections per machine, which are continuously opened and closed. Still, it is useful as a baseline and should work fine with smaller setups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Locked
