[Discuss] Read from S3 using AvroInputFormat

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Discuss] Read from S3 using AvroInputFormat

Vinay Patil
Hi,

I am able to read avro file stored in S3 using AvroInputFormat by providing
the full path.

Is there a way I can read all the folders inside the bucket using existing
Flink API.

I know we can do this using the S3 REST Service, but I am more interested
to do it in Flink (may be using Flink Dataset API for historical
re-processing)

Regards,
Vinay Patil
Reply | Threaded
Open this post in threaded view
|

Re: [Discuss] Read from S3 using AvroInputFormat

Till Rohrmann
Hi Vinay,

have you tried only specifying the directory path you want to read [1]?

I think in the future these kind of questions should be posted to the user
mailing list [hidden email]. The dev mailing list should be used for
discussions about Flink's development.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html#recursive-traversal-of-the-input-path-directory

Cheers,
Till

On Fri, Jul 8, 2016 at 1:50 PM, Vinay Patil <[hidden email]> wrote:

> Hi,
>
> I am able to read avro file stored in S3 using AvroInputFormat by providing
> the full path.
>
> Is there a way I can read all the folders inside the bucket using existing
> Flink API.
>
> I know we can do this using the S3 REST Service, but I am more interested
> to do it in Flink (may be using Flink Dataset API for historical
> re-processing)
>
> Regards,
> Vinay Patil
>
Reply | Threaded
Open this post in threaded view
|

Re: [Discuss] Read from S3 using AvroInputFormat

Vinay Patil
I was not aware that there is separate mailing list for discussions.
From next time I will use that mailing list for my queries.

Thanks Till for that information.

Regards,
Vinay Patil

On Fri, Jul 8, 2016 at 5:38 PM, Till Rohrmann <[hidden email]> wrote:

> Hi Vinay,
>
> have you tried only specifying the directory path you want to read [1]?
>
> I think in the future these kind of questions should be posted to the user
> mailing list [hidden email]. The dev mailing list should be used
> for
> discussions about Flink's development.
>
> [1]
>
> https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html#recursive-traversal-of-the-input-path-directory
>
> Cheers,
> Till
>
> On Fri, Jul 8, 2016 at 1:50 PM, Vinay Patil <[hidden email]>
> wrote:
>
> > Hi,
> >
> > I am able to read avro file stored in S3 using AvroInputFormat by
> providing
> > the full path.
> >
> > Is there a way I can read all the folders inside the bucket using
> existing
> > Flink API.
> >
> > I know we can do this using the S3 REST Service, but I am more interested
> > to do it in Flink (may be using Flink Dataset API for historical
> > re-processing)
> >
> > Regards,
> > Vinay Patil
> >
>