Re: NoSuchMethodError: org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader

[Guowei Ma]

Sorry, I check the code find that '
org/apache/flink/fs/s3presto/common/HadoopConfigLoader'' is shaded on
purpose.
So could you check whether there is a old "presto" jar in your lib
directory?
Best,
Guowei


Claude Murad <[hidden email]> 于2020年6月10日周三 上午10:06写道：

> Thanks for your reply.  The
> org.apache.flink.fs.s3presto.common.HadoopConfigLoader is not in the jar.
> Did you mean to remove the S3FileSystemFactory if the class doesn't exist?
> If so, how will it work without the S3FileSystemFactory?  Also, this is the
> jar that was packaged in the Flink docker image.
>
> On Tue, Jun 9, 2020 at 9:46 PM Guowei Ma <[hidden email]> wrote:
>
>> Hi,
>> In 1.10 there is no
>> 'Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader' . So I think
>> there might be a legacy S3FileSystemFactory in your jar. You could check
>> whether there is a 'org.apache.flink.fs.s3presto.common.HadoopConfigLoader'
>> in your jar or not. If there is one you could remove the
>> old S3FileSystemFactory and try again.
>>
>> Btw I think you might not copy both flink-s3-fs-hadoop-1.10.0
>> and  flink-s3-fs-presto-1.10.0.jar to the same plugin dir. [1]
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/filesystems/s3.html#hadooppresto-s3-file-systems-plugins
>> Best,
>> Guowei
>>
>>
>> Claude Murad <[hidden email]> 于2020年6月10日周三 上午4:06写道：
>>
>>> Hello,
>>>
>>> I'm trying to upgrade Flink from 1.7 to 1.10 retaining our Hadoop
>>> integration.  I copied the jar
>>> file flink-shaded-hadoop-2-uber-2.7.5-10.0.jar into /opt/flink/lib.  I also
>>> copied the files flink-s3-fs-hadoop-1.10.0.jar and
>>> flink-s3-fs-presto-1.10.0.jar into /opt/flink/plugins/s3 folder.  The error
>>> below occurs after deploying and launching docker image 1.10.0-scala_2.11.
>>> I saw that S3FileSystemFactory.java is now importing
>>> org.apache.flink.runtime.util.HadoopConfigLoader instead of
>>> org.apache.flink.fs.s3.common.HadoopConfigLoader which is how it was
>>> before.  I see the jar file flink-dist_2.11-1.10.0.jar contains
>>> the org.apache.flink.runtime.util.HadoopConfigLoader and it is under the
>>> folder /opt/flink/lib.  Any ideas on how to resolve this error?  Any help
>>> would be greatly appreciated, thank you.
>>>
>>>
>>> ERROR org.apache.flink.core.fs.FileSystem                           -
>>> Failed to load a file system via services
>>> java.util.ServiceConfigurationError:
>>> org.apache.flink.core.fs.FileSystemFactory: Provider
>>> org.apache.flink.fs.s3presto.S3PFileSystemFactory could not be instantiated
>>> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>>> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>>> at
>>> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>>> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>>> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>>> at
>>> org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1024)
>>> at
>>> org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1006)
>>> at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:303)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:194)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:164)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
>>> at
>>> org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main(StandaloneSessionClusterEntrypoint.java:64)
>>> Caused by: java.lang.NoSuchMethodError:
>>> org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(Ljava/lang/String;Lorg/apache/flink/fs/s3presto/common/HadoopConfigLoader;)V
>>> at
>>> org.apache.flink.fs.s3presto.S3FileSystemFactory.<init>(S3FileSystemFactory.java:50)
>>> at
>>> org.apache.flink.fs.s3presto.S3PFileSystemFactory.<init>(S3PFileSystemFactory.java:24)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>> at java.lang.Class.newInstance(Class.java:442)
>>> at
>>> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>>>
>>