Yang Wang created FLINK-13837:
---------------------------------
Summary: Support --files and --libjars arguments in flink run command line
Key: FLINK-13837
URL:
https://issues.apache.org/jira/browse/FLINK-13837 Project: Flink
Issue Type: New Feature
Components: Command Line Client
Reporter: Yang Wang
Currently we could use the following codes to register a cached file and then get it in the task. We hope it could be done more easier by --files command option, such as —files [file:///tmp/test_data].
*final* StreamExecutionEnvironment env = StreamExecutionEnvironment._getExecutionEnvironment_();
env.registerCachedFile(inputFile.toString(), *"test_data"*, *false*);
For a jar, we could build a fat jar including our codes and all dependencies . It is better to add --libjars command option to support transfer dependencies.
What’s the difference between --files&—libjars and -yt?
* Option -yt is used when submitting job to YARN cluster, and all files will be distributed by YARN distributed cache. It will be shared by all jobs in the flink cluster.
* Option --libjars is used for flink job, and all files will be distributed by blob server. It is only accessible for the specific job.
The new added command options are as follows.
--files Attach custom files for job. Directory
could not be supported. Use ',' to
separate multiple files. The files
could be in local file system or
distributed file system. Use URI
schema to specify which file system
the file belongs. If schema is
missing, would try to get the file in
local file system. Use '#' after the
file path to specify retrieval key in
runtime. (eg: --file
file:///tmp/a.txt#file_key,hdfs:///$na
menode_address/tmp/b.txt)
--libjars Attach custom library jars for job.
Directory could not be supported. Use
',' to separate multiple jars. The
jars could be in local file system or
distributed file system. Use URI
schema to specify which file system
the jar belongs. If schema is missing,
would try to get the jars in local
file system. (eg: --libjars
file:///tmp/dependency1.jar,hdfs:///$n
amenode_address/tmp/dependency2.jar)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
Locked
