Hi,
the SequenceFileWriter and the AvroKeyValueSinkWriter both support
compressed outputs. Apart from that, I'm not aware of any other Writers
which support compression. Maybe you could use these two Writers as a
guiding example. Alternatively, you could try to extend the
StreamWriterBase and wrapping the outStream into a GZIPOutputStream.
Cheers,
Till
On Wed, Mar 28, 2018 at 1:59 AM,
[hidden email] <
[hidden email]> wrote:
> I want to upload a compressed file (gzip preferrably) using the Bucketing
> Sink. What is the best way to do this? Would I have to implement my own
> Writer that does the compression? Has anyone done something similar?
>