Till Rohrmann created FLINK-2726:
------------------------------------
Summary: Harmonise Yarn configuration methods to use a single configuration for all components
Key: FLINK-2726
URL:
https://issues.apache.org/jira/browse/FLINK-2726 Project: Flink
Issue Type: Improvement
Components: YARN Client
Reporter: Till Rohrmann
Currently, the {{FlinkYarnClient}} uses different configuration objects to initialize the {{ApplicationClient}} and the {{ApplicationMaster}}.
The {{ApplicationMaster}} is started with a configuration which is read from disk by specifying its path. The configuration file is uploaded to HDFS to make it accessible. Additionally, command line properties are encoded in an environment variable which will also be transferred to the started Yarn containers.
The {{ApplicationClient}} is started with a {{Configuration}} object which is given via a setter to the {{FlinkYarnClient}}. The command line options will be added to this configuration object. The configuration object does not have to be the same as the configuration object which is read from disk. This can lead to serious problems when, for example, the {{akka.framesize}} has to be consistently configured. At the moment, this only works because the configuration object is created in the {{FlinkYarnSessionCli}} from the same file which is also uploaded to HDFS.
I propose to only use a single mean for configuring the different Yarn components. One could, for example, only set the {{Configuration}} object which is then written to HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Locked
