Wei Zhong created FLINK-14590:
---------------------------------
Summary: Unify the working directory of Java process and Python process when submitting python jobs via "flink run -py"
Key: FLINK-14590
URL:
https://issues.apache.org/jira/browse/FLINK-14590 Project: Flink
Issue Type: Bug
Components: API / Python
Reporter: Wei Zhong
Assume we enter this flink directory with following structure:
{code:java}
flink/
bin/
flink
pyflink-shell.sh
python-gateway-server.sh
...
bad_case/
word_count.py
data.txt
lib/...
opt/...{code}
And the word_count.py has such a piece of code:
{code:java}
t_config = TableConfig()
env = StreamExecutionEnvironment.get_execution_environment()
t_env = StreamTableEnvironment.create(env, t_config)
env._j_stream_execution_environment.registerCachedFile("data", "bad_case/data.txt")
with open("bad_case/data.txt", "r") as f:
content = f.read()
elements = [(word, 1) for word in content.split(" ")]
t_env.from_elements(elements, ["word", "count"]){code}
Then we enter the "flink" directory and run:
{code:java}
bin/flink run -py bad_case/word_count.py
{code}
The program will fail at the line of "with open("bad_case/data.txt", "r") as f:".
It is because the working directory of Java process is current directory but the working directory of Python process is a temporary directory.
So there is no problem when relative path is used in the api call to java process. But if relative path is used in other place such as native file access, it will fail, because the working directory of python process has been change to a temporary directory that is not known to users.
I think it will cause some confusion for users, especially after we support dependency management. It will be great if we unify the working directory of Java process and Python process.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
