[jira] [Created] (FLINK-11459) Presto S3 does not show errors due to missing credentials with minio

[Shang Yuanchun (Jira)]

Nico Kruber created FLINK-11459:
-----------------------------------

             Summary: Presto S3 does not show errors due to missing credentials with minio
                 Key: FLINK-11459
                 URL: https://issues.apache.org/jira/browse/FLINK-11459
             Project: Flink
          Issue Type: Bug
          Components: filesystem-connector
    Affects Versions: 1.6.2
            Reporter: Nico Kruber


It seems that when using minio for S3-like storage and with mis-configurations such as missing (maybe also wrong) credentials gets into a failing state but with no reason for it:
{code}
...
2019-01-29 15:43:27,676 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: taskmanager.heap.mb, 353
2019-01-29 15:43:27,738 INFO  org.apache.flink.configuration.GlobalConfiguration           [] - Loading configuration property: jobmanager.heap.mb, 429
2019-01-29 15:43:27,758 INFO  org.apache.flink.api.java.ExecutionEnvironment               [] - The job has 0 registered types and 0 default Kryo serializers
2019-01-29 15:43:29,943 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
2019-01-29 15:43:29,956 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2115.551],
2019-01-29 15:43:31,946 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
2019-01-29 15:43:31,958 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[3610.417],
2019-01-29 15:43:33,954 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
2019-01-29 15:43:33,963 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2917.786],
2019-01-29 15:43:36,133 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
2019-01-29 15:43:36,156 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2029.473],
2019-01-29 15:43:38,142 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
2019-01-29 15:43:38,164 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2092.878],
2019-01-29 15:43:42,181 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
2019-01-29 15:43:42,186 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2011.204],
2019-01-29 15:43:44,262 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
2019-01-29 15:43:44,276 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency    [] - ClientExecuteTime=[2024.312],
2019-01-29 15:43:44,585 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2019-01-29 15:43:44,628 INFO  org.apache.flink.runtime.blob.TransientBlobCache             [] - Shutting down BLOB cache
2019-01-29 15:43:44,661 INFO  org.apache.flink.runtime.blob.BlobServer                     [] - Stopped BLOB server at 0.0.0.0:6124
{code}

With AWS S3, it is actually printing an exception instead:
{code}
2019-01-29 19:24:39,968 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2019-01-29 19:24:39,990 INFO  org.apache.flink.api.java.ExecutionEnvironment                - The job has 0 registered types and 0 default Kryo serializers
2019-01-29 19:24:43,117 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
2019-01-29 19:24:43,118 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2049.826],
2019-01-29 19:24:46,215 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
2019-01-29 19:24:46,216 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2004.182],
2019-01-29 19:24:50,384 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2003.15], CredentialsRequestTime=[2002.803],
2019-01-29 19:24:50,384 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2004.308],
2019-01-29 19:24:56,691 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2002.596], CredentialsRequestTime=[2002.45],
2019-01-29 19:24:56,691 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2003.177],
2019-01-29 19:25:07,058 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2003.26], CredentialsRequestTime=[2002.948],
2019-01-29 19:25:07,058 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2004.175],
2019-01-29 19:25:25,472 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2001.772], CredentialsRequestTime=[2001.611],
2019-01-29 19:25:25,473 INFO  org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency     - ClientExecuteTime=[2002.873],
2019-01-29 19:25:25,475 ERROR org.apache.flink.api.common.io.DelimitedInputFormat           - Unexpected problem while getting the file statistics for files '[s3://flink/LICENSE.gz]': Unable to load credentials from service endpoint
org.apache.flink.fs.s3presto.shaded.com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:180)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:159)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228)
        at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559)
        at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
        at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556)
        at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307)
        at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:85)
        at org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:526)
        at org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:505)
        at org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:356)
        at org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:47)
        at org.apache.flink.optimizer.dag.DataSourceNode.computeOperatorSpecificDefaultEstimates(DataSourceNode.java:166)
        at org.apache.flink.optimizer.dag.OptimizerNode.computeOutputEstimates(OptimizerNode.java:589)
        at org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:61)
        at org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:32)
        at org.apache.flink.optimizer.dag.DataSourceNode.accept(DataSourceNode.java:250)
        at org.apache.flink.optimizer.dag.SingleInputNode.accept(SingleInputNode.java:515)
        at org.apache.flink.optimizer.dag.DataSinkNode.accept(DataSinkNode.java:248)
        at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:478)
        at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:399)
        at org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:51)
        at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817)
        at test.FlinkReadS3Test.main(FlinkReadS3Test.java:36)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
        at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
        at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
        at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
        at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$10(JarRunHandler.java:226)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
        at sun.net.www.http.HttpClient.New(HttpClient.java:339)
        at sun.net.www.http.HttpClient.New(HttpClient.java:357)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:47)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:106)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:77)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:156)
        at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121)
        ... 52 more
...
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)