ChengWei Ye created FLINK-17858:
-----------------------------------
Summary: Yarn mode, windows and linux environment should be interlinked
Key: FLINK-17858
URL:
https://issues.apache.org/jira/browse/FLINK-17858 Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Affects Versions: 1.10.0
Reporter: ChengWei Ye
my flink version: 1.10.0
my yarn version: 2.7.6
When I use windows idea to try to submit a job to Linux's yarn cluster using java, am container throws an exception "Error: Could not find or load main class org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint".
Then find out that the CLASSPATH and _FLINK_CLASSPATH delimiter of the container startup script "launch_container.sh" are different, windows is ";", linux is ":", so the container cannot get the dependent packages.
I think the operating environment should be left to Yarn to judge.
Locate in the flink-yarn module
1. The startAppMaster method of org.apache.flink.yarn.YarnClusterDescriptor
Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
Yarn will replace special characters according to the environment, but this only solves _FLINK_CLASSPATH
{code:java}
private ApplicationReport startAppMaster(.....) {
.....
StringBuilder classPathBuilder = new StringBuilder();
if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
for (String userClassPath : userClassPaths) {
// here
classPathBuilder.append(userClassPath).append(File.pathSeparator);
}
}
for (String classPath : systemClassPaths) {
// here
classPathBuilder.append(classPath).append(File.pathSeparator);
}
// Setup jar for ApplicationMaster
.....
// here
classPathBuilder.append(flinkJarPath.getName()).append(File.pathSeparator);
.....
}{code}
2. The addToEnvironment method of org.apache.flink.yarn.Utils
Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
Can solve the CLASSPATH
{code:java}
public static void addToEnvironment(Map<String, String> environment,
String variable, String value) {
String val = environment.get(variable);
if (val == null) {
val = value;
} else {
// here
val = val + File.pathSeparator + value;
}
environment.put(StringInterner.weakIntern(variable),
StringInterner.weakIntern(val));
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
