[DISCUSS] -- Need to be able to access multiple s3 buckets with different authentication from within a single job

[Dave Brewster]

Currently there doesn’t seem to be a way to do this.  Am I correct in that?  I guess one could register multiple implementations each with their own scheme but that seems somewhat hacky.

It would be nice if the registration of the filesystem was done when the DataSet (or DataStream) was defined and not have them preregistered in the system.  I’ve looked at the code and it doesn’t seem too terribly difficult.

I was wondering if others have a need for this as well?  

Thanks,

Dave Brewster

[Stephan Ewen]

Sorry for the very late reply.

I would try to see if you can simply pass in an instance of filesystem into
the source / sink you are using, rather than relying on getting the file
system via the scheme.

The only point where you probably need to accept the file system by scheme
is for checkpoints, but for sources / sinks it should be possible to pass
the FS as an argument. If that is missing (a constructor that accepts a FS
factory or so), then this makes sense to add.


On Thu, Sep 6, 2018 at 3:07 AM Dave Brewster <[hidden email]>
wrote:

>
> Currently there doesn’t seem to be a way to do this.  Am I correct in
> that?  I guess one could register multiple implementations each with their
> own scheme but that seems somewhat hacky.
>
> It would be nice if the registration of the filesystem was done when the
> DataSet (or DataStream) was defined and not have them preregistered in the
> system.  I’ve looked at the code and it doesn’t seem too terribly difficult.
>
> I was wondering if others have a need for this as well?
>
> Thanks,
>
> Dave Brewster
>
>