[jira] [Created] (FLINK-19798) Support creating hive table input splits in parallel

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (FLINK-19798) Support creating hive table input splits in parallel

Shang Yuanchun (Jira)
Zou created FLINK-19798:
---------------------------

             Summary: Support creating hive table input splits in parallel
                 Key: FLINK-19798
                 URL: https://issues.apache.org/jira/browse/FLINK-19798
             Project: Flink
          Issue Type: Improvement
            Reporter: Zou


For now, we could only creating hive table input splits serially.  
HiveTableInputFormat::createInputSplits cost 40s~60s for a table with 1500 partition in my tests. IMO, we could support creating splits in parallel, which will save most of the time to do this.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)