[jira] [Created] (FLINK-3964) Job submission times out with recursive.file.enumeration

[Shang Yuanchun (Jira)]

Juho Autio created FLINK-3964:
---------------------------------

             Summary: Job submission times out with recursive.file.enumeration
                 Key: FLINK-3964
                 URL: https://issues.apache.org/jira/browse/FLINK-3964
             Project: Flink
          Issue Type: Bug
            Reporter: Juho Autio


When using "recursive.file.enumeration" with a big enough folder structure to list, flink batch job fails right at the beginning because of a timeout.

h2. Problem details

We get this error: {{Communication with JobManager failed: Job submission to the JobManager timed out}}.

The code we have is basically this:

{code}
val env = ExecutionEnvironment.getExecutionEnvironment

val parameters = new Configuration

// set the recursive enumeration parameter
parameters.setBoolean("recursive.file.enumeration", true)

val parameter = ParameterTool.fromArgs(args)

val input_data_path : String = parameter.get("input_data_path", null )

val data : DataSet[(Text,Text)] = env.readSequenceFile(classOf[Text], classOf[Text], input_data_path)
.withParameters(parameters)

data.first(10).print
{code}

If we set {{input_data_path}} parameter to {{s3n://bucket/path/date=*/}} it times out. If we use a more restrictive pattern like {{s3n://bucket/path/date=20160523/}}, it doesn't time out.

To me it seems that time taken to list files shouldn't cause any timeouts on job submission level.

For us this was "fixed" by adding {{akka.client.timeout: 600 s}} in {{flink-conf.yaml}}, but I wonder if the timeout would still occur if we have even more files to list?

----

P.S. Is there any way to set {{akka.client.timeout}} when calling {{bin/flink run}} instead of editing {{flink-conf.yaml}}. I tried to add it as a {{-yD}} flag but couldn't get it working.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)