What does setExecutionConfig(new ExecutionConfig()) in the constructor of JobGraph for?

[ZhangWei]

Hello:


    I was recently study the code about how the JobGraph generates. I noticed in the constructor of StreamingJobGraphGenerator, it constructs a JobGraph. And in the constructor of JobGraph, it called setExecutionConfig(new ExecutionConfig()) which serialized a default ExecutionConfig. The comment said "Further modification of the referenced ExecutionConfig object will not affect this serialized copy".
    While at the end of StreamingJobGraphGenerator#createJobGraph(), it called jobGraph.setExecutionConfig(streamGraph.getExecutionConfig()) again, which sets the ExecutionConfig contains the parameters we have changed. I think that does the right thing.
   So I wonder what the setExecutionConfig() in the constructor of JobGraph for? It seems meaningless for me.


Best regards!

[Chesnay Schepler-3]

It just initializes the field to ensure it is set when it is
deserialized server-side.
There is neither a guarantee nor requirement that the APIs must set the
ExecutionConfig at least once, so this ensures that on the server-side
the execution config is never null.

On 3/26/2021 11:18 AM, ZhangWei wrote:

> Hello:
>
>
>      I was recently study the code about how the JobGraph generates. I noticed in the constructor of StreamingJobGraphGenerator, it constructs a JobGraph. And in the constructor of JobGraph, it called setExecutionConfig(new ExecutionConfig()) which serialized a default ExecutionConfig. The comment said "Further modification of the referenced ExecutionConfig object will not affect this serialized copy".
>      While at the end of StreamingJobGraphGenerator#createJobGraph(), it called jobGraph.setExecutionConfig(streamGraph.getExecutionConfig()) again, which sets the ExecutionConfig contains the parameters we have changed. I think that does the right thing.
>     So I wonder what the setExecutionConfig() in the constructor of JobGraph for? It seems meaningless for me.
>
>
> Best regards!
>
>
>
>
>
>
>

[ZhangWei]

Hi Chesnay:
If the server side expects the serialized execution config never be null, I think it just equals that we must setExecutionConfig somewhere either in the constructor or after the jobGraph has been instantiated at least once.
The current implementation sets ExecutionConfig in the constructor of JobGraph in case the default ExecutionConfig works well, so that we do not need to setExecutionConfig again. But this is really rare. Often we set the Parallelism/maxParallelism/snapshot-compression/restartstrategy/custom-serializers etcs from code/flink-conf.yaml/shell args. The default values of executionConfig often do not work well.
   So, in StreamingJobGraphGenerator#createJobGraph(), we set again the config from StreamGraph.getExecutionConfig() which contains the parameters we set. I think the setExecutionConfig method in the constructor is often unnecessary. it is often overwritten by the parameters we set.





















At 2021-03-26 18:34:22, "Chesnay Schepler" <[hidden email]> wrote:

>It just initializes the field to ensure it is set when it is
>deserialized server-side.
>There is neither a guarantee nor requirement that the APIs must set the
>ExecutionConfig at least once, so this ensures that on the server-side
>the execution config is never null.
>
>On 3/26/2021 11:18 AM, ZhangWei wrote:
>> Hello:
>>
>>
>>      I was recently study the code about how the JobGraph generates. I noticed in the constructor of StreamingJobGraphGenerator, it constructs a JobGraph. And in the constructor of JobGraph, it called setExecutionConfig(new ExecutionConfig()) which serialized a default ExecutionConfig. The comment said "Further modification of the referenced ExecutionConfig object will not affect this serialized copy".
>>      While at the end of StreamingJobGraphGenerator#createJobGraph(), it called jobGraph.setExecutionConfig(streamGraph.getExecutionConfig()) again, which sets the ExecutionConfig contains the parameters we have changed. I think that does the right thing.
>>     So I wonder what the setExecutionConfig() in the constructor of JobGraph for? It seems meaningless for me.
>>
>>
>> Best regards!
>>
>>
>>
>>
>>
>>
>>