[jira] [Created] (FLINK-14396) Implement rudimentary non-blocking network output

[Shang Yuanchun (Jira)]

zhijiang created FLINK-14396:
--------------------------------

             Summary: Implement rudimentary non-blocking network output
                 Key: FLINK-14396
                 URL: https://issues.apache.org/jira/browse/FLINK-14396
             Project: Flink
          Issue Type: Task
          Components: Runtime / Network
            Reporter: zhijiang
            Assignee: zhijiang
             Fix For: 1.10.0


Considering the mailbox model and unaligned checkpoints requirements in future, task network output should be non-blocking. In other words, as long as output is available, it should never block for a subsequent/future single record write.

In the first version, we only implement the non-blocking output for the most regular case, and do not solve the following cases which still keep the previous behavior.
 * Big record which might span multiple buffers
 * Flatmap-like operators which might emit multiple records in every process
 * Broadcast watermark which might request multiple buffers at a time

The solution is providing the RecordWriter#isAvailable method and respective LocalBufferPool#isAvailable for judging the output beforehand. As long as there is at-least one available buffer in LocalBufferPool, the RecordWriter is available for network output in most cases.  This doesn’t include runtime handling of this non-blocking and availability behavior in StreamInputProcessor

Note: It requires the minimum number of buffers in output LocalBufferPool adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of backpressure future.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)