Stephan Ewen created FLINK-19221:
------------------------------------
Summary: Exploit LocatableFileStatus from Hadoop
Key: FLINK-19221
URL:
https://issues.apache.org/jira/browse/FLINK-19221 Project: Flink
Issue Type: Improvement
Components: Connectors / Hadoop Compatibility
Affects Versions: 1.11.1
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.12.0
When the HDFS Client returns a {{FileStatus}} (description of a file) it sometimes returns a {{LocatedFileStatus}} which already contains all the {{BlockLocation}} information.
We should expose this on the Flink side, because it may save is a lot of RPC calls to the name node. The file enumerators often request block locations for all files, currently doing an RPC call for each file.
When the FileStatus obtained from listing the directory (or getting details for a file) already has all the block locations, we can save the extra RPC call per file.
The suggested implementation is as follows:
1. We introduce a {{LocatedInputSplit}} in Flink that we integrate with the built-in LocalFileSystem
2. We integrate this with the HadoopFileSystems by creating a Flink {{LocatedInputSplit}} whenever the underlying file system created a {{Hadoop LocatedInputSplit}}
3. As a safety net, the FS methods to access block information check whether the presented file status already contains the block information and return that information directly.
Steps one and two are for simplification of FileSystem users (no need to ask for extra info if it is available).
Step three is the transparent shortcut that all applications get even if they do not explicitly use the {{LocatedInputSplit}} and keep calling {{FileSystem.getBlockLocations()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
