zoucao created FLINK-20827:
------------------------------
Summary: Just read record correlating to join key in FilesystemLookUpFunc
Key: FLINK-20827
URL:
https://issues.apache.org/jira/browse/FLINK-20827 Project: Flink
Issue Type: Improvement
Components: Connectors / FileSystem
Reporter: zoucao
When using Temporal table join, all hive tables' records will be loaded into cache. But sometimes, the size of hive temporal table is larger than expected, and users can't know exactly how big it is in memory. In this situation, some error will occur, for example, `GC overhead limit exceeded`, `the heartbeat of TaskManager timeout (caused by gc)`.
Maybe we can optimize the number of records readed from hive table? If the upstream records can be hashed only by using `Join key`, then we only need to load the part of records into cache, whose value of join key after being hashed, is equal to one fixed hash value. If it can be done, the whole table can be divided by the number of parallelism. I don't know whether it could come true In the upstream under the existing framework, but It is easy to support in `FileSystemLookupFunction`
If not, we can add some log to tell others the size of cache to help them to set MemorySize or other parameter of TM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
