[jira] [Created] (FLINK-13938) Use yarn public distributed cache to speed up containers launch

[Shang Yuanchun (Jira)]

Yang Wang created FLINK-13938:
---------------------------------

             Summary: Use yarn public distributed cache to speed up containers launch
                 Key: FLINK-13938
                 URL: https://issues.apache.org/jira/browse/FLINK-13938
             Project: Flink
          Issue Type: New Feature
            Reporter: Yang Wang


By default, the LocalResourceVisibility is APPLICATION, so they will be downloaded only once and shared for all taskmanager containers of a same application in the same node. However, different applications will have to download all jars every time, including the flink-dist.jar. I think we could use the yarn public cache to eliminate the unnecessary jars downloading and make launching container faster.

 

How to use the shared lib feature?
 # Upload a copy of flink release binary to hdfs.
 # Use the -ysl argument to specify the shared lib

{code:java}
./bin/flink run -d -m yarn-cluster -p 20 -ysl hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
 

-ysl,--yarnsharedLib <path>          Upload a copy of flink lib beforehand

                                                          and specify the path to use public

                                                          visibility feature of YARN NodeManager

                                                          localizing resources.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)