Parquet Article / Tutorial

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Parquet Article / Tutorial

Felix Neutatz
Hi everybody,

I am working currently on a tutorial/article about how/when/why to use
Parquet on Flink.

You can find the pdf version here:
https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf

The git repository with all the code examples can be found here:
https://github.com/FelixNeutatz/parquet-flinktacular/

What do you think about it? I am happy about every feedback I can get :)

Have a nice Sunday,

Felix
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Stephan Ewen
Wow, very nice work!

It looks impressive at first sight, no comments :-)

Just one question: Do you want this to be a standalone tutorial, or are you
thinking of putting this into the Flink wiki, or some code even into
"flink-contrib" ?

Stephan


On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <[hidden email]>
wrote:

> Hi everybody,
>
> I am working currently on a tutorial/article about how/when/why to use
> Parquet on Flink.
>
> You can find the pdf version here:
>
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
>
> The git repository with all the code examples can be found here:
> https://github.com/FelixNeutatz/parquet-flinktacular/
>
> What do you think about it? I am happy about every feedback I can get :)
>
> Have a nice Sunday,
>
> Felix
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Felix Neutatz
The intention was to post it on the blog, but if you think it would better
fit into the wiki, that would be also fine :)

About the code: I have not thought about putting it to contrib-package, but
I can. What do you think is suitable for flink-contrib?

Best regards,

Felix

2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]>:

> Wow, very nice work!
>
> It looks impressive at first sight, no comments :-)
>
> Just one question: Do you want this to be a standalone tutorial, or are you
> thinking of putting this into the Flink wiki, or some code even into
> "flink-contrib" ?
>
> Stephan
>
>
> On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <[hidden email]>
> wrote:
>
> > Hi everybody,
> >
> > I am working currently on a tutorial/article about how/when/why to use
> > Parquet on Flink.
> >
> > You can find the pdf version here:
> >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> >
> > The git repository with all the code examples can be found here:
> > https://github.com/FelixNeutatz/parquet-flinktacular/
> >
> > What do you think about it? I am happy about every feedback I can get :)
> >
> > Have a nice Sunday,
> >
> > Felix
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Kostas Tzoumas-2
Looks very nice! Would love to see a blog post on that!

On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <[hidden email]>
wrote:

> The intention was to post it on the blog, but if you think it would better
> fit into the wiki, that would be also fine :)
>
> About the code: I have not thought about putting it to contrib-package, but
> I can. What do you think is suitable for flink-contrib?
>
> Best regards,
>
> Felix
>
> 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]>:
>
> > Wow, very nice work!
> >
> > It looks impressive at first sight, no comments :-)
> >
> > Just one question: Do you want this to be a standalone tutorial, or are
> you
> > thinking of putting this into the Flink wiki, or some code even into
> > "flink-contrib" ?
> >
> > Stephan
> >
> >
> > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <[hidden email]>
> > wrote:
> >
> > > Hi everybody,
> > >
> > > I am working currently on a tutorial/article about how/when/why to use
> > > Parquet on Flink.
> > >
> > > You can find the pdf version here:
> > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > >
> > > The git repository with all the code examples can be found here:
> > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > >
> > > What do you think about it? I am happy about every feedback I can get
> :)
> > >
> > > Have a nice Sunday,
> > >
> > > Felix
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Fabian Hueske-2
Very nice article!
How about adding the full article to the wiki and having a shorter version
as a blog post (with a link to the wiki)?
Adding the code to contrib would be great!

2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]>:

> Looks very nice! Would love to see a blog post on that!
>
> On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <[hidden email]>
> wrote:
>
> > The intention was to post it on the blog, but if you think it would
> better
> > fit into the wiki, that would be also fine :)
> >
> > About the code: I have not thought about putting it to contrib-package,
> but
> > I can. What do you think is suitable for flink-contrib?
> >
> > Best regards,
> >
> > Felix
> >
> > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]>:
> >
> > > Wow, very nice work!
> > >
> > > It looks impressive at first sight, no comments :-)
> > >
> > > Just one question: Do you want this to be a standalone tutorial, or are
> > you
> > > thinking of putting this into the Flink wiki, or some code even into
> > > "flink-contrib" ?
> > >
> > > Stephan
> > >
> > >
> > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <[hidden email]>
> > > wrote:
> > >
> > > > Hi everybody,
> > > >
> > > > I am working currently on a tutorial/article about how/when/why to
> use
> > > > Parquet on Flink.
> > > >
> > > > You can find the pdf version here:
> > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > >
> > > > The git repository with all the code examples can be found here:
> > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > >
> > > > What do you think about it? I am happy about every feedback I can get
> > :)
> > > >
> > > > Have a nice Sunday,
> > > >
> > > > Felix
> > > >
> > >
> >
>
mxm
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

mxm
Hi Felix,

Very nice informative read.

+1 for a short blog post and a full version in the wiki.
+1 for putting this into flink-contrib


On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]> wrote:

> Very nice article!
> How about adding the full article to the wiki and having a shorter version
> as a blog post (with a link to the wiki)?
> Adding the code to contrib would be great!
>
> 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]>:
>
> > Looks very nice! Would love to see a blog post on that!
> >
> > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <[hidden email]>
> > wrote:
> >
> > > The intention was to post it on the blog, but if you think it would
> > better
> > > fit into the wiki, that would be also fine :)
> > >
> > > About the code: I have not thought about putting it to contrib-package,
> > but
> > > I can. What do you think is suitable for flink-contrib?
> > >
> > > Best regards,
> > >
> > > Felix
> > >
> > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]>:
> > >
> > > > Wow, very nice work!
> > > >
> > > > It looks impressive at first sight, no comments :-)
> > > >
> > > > Just one question: Do you want this to be a standalone tutorial, or
> are
> > > you
> > > > thinking of putting this into the Flink wiki, or some code even into
> > > > "flink-contrib" ?
> > > >
> > > > Stephan
> > > >
> > > >
> > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi everybody,
> > > > >
> > > > > I am working currently on a tutorial/article about how/when/why to
> > use
> > > > > Parquet on Flink.
> > > > >
> > > > > You can find the pdf version here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > > >
> > > > > The git repository with all the code examples can be found here:
> > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > > >
> > > > > What do you think about it? I am happy about every feedback I can
> get
> > > :)
> > > > >
> > > > > Have a nice Sunday,
> > > > >
> > > > > Felix
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Henry Saputra
+1 to the idea.

Awesome work, Felix

On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]> wrote:

> Hi Felix,
>
> Very nice informative read.
>
> +1 for a short blog post and a full version in the wiki.
> +1 for putting this into flink-contrib
>
>
> On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]
> <javascript:;>> wrote:
>
> > Very nice article!
> > How about adding the full article to the wiki and having a shorter
> version
> > as a blog post (with a link to the wiki)?
> > Adding the code to contrib would be great!
> >
> > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]
> <javascript:;>>:
> >
> > > Looks very nice! Would love to see a blog post on that!
> > >
> > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <[hidden email]
> <javascript:;>>
> > > wrote:
> > >
> > > > The intention was to post it on the blog, but if you think it would
> > > better
> > > > fit into the wiki, that would be also fine :)
> > > >
> > > > About the code: I have not thought about putting it to
> contrib-package,
> > > but
> > > > I can. What do you think is suitable for flink-contrib?
> > > >
> > > > Best regards,
> > > >
> > > > Felix
> > > >
> > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
> <javascript:;>>:
> > > >
> > > > > Wow, very nice work!
> > > > >
> > > > > It looks impressive at first sight, no comments :-)
> > > > >
> > > > > Just one question: Do you want this to be a standalone tutorial, or
> > are
> > > > you
> > > > > thinking of putting this into the Flink wiki, or some code even
> into
> > > > > "flink-contrib" ?
> > > > >
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> > [hidden email] <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > Hi everybody,
> > > > > >
> > > > > > I am working currently on a tutorial/article about how/when/why
> to
> > > use
> > > > > > Parquet on Flink.
> > > > > >
> > > > > > You can find the pdf version here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > > > >
> > > > > > The git repository with all the code examples can be found here:
> > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > > > >
> > > > > > What do you think about it? I am happy about every feedback I can
> > get
> > > > :)
> > > > > >
> > > > > > Have a nice Sunday,
> > > > > >
> > > > > > Felix
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Robert Metzger
+1 for a blog post

On Tue, Apr 7, 2015 at 5:17 PM, Henry Saputra <[hidden email]>
wrote:

> +1 to the idea.
>
> Awesome work, Felix
>
> On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]> wrote:
>
> > Hi Felix,
> >
> > Very nice informative read.
> >
> > +1 for a short blog post and a full version in the wiki.
> > +1 for putting this into flink-contrib
> >
> >
> > On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]
> > <javascript:;>> wrote:
> >
> > > Very nice article!
> > > How about adding the full article to the wiki and having a shorter
> > version
> > > as a blog post (with a link to the wiki)?
> > > Adding the code to contrib would be great!
> > >
> > > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]
> > <javascript:;>>:
> > >
> > > > Looks very nice! Would love to see a blog post on that!
> > > >
> > > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <
> [hidden email]
> > <javascript:;>>
> > > > wrote:
> > > >
> > > > > The intention was to post it on the blog, but if you think it would
> > > > better
> > > > > fit into the wiki, that would be also fine :)
> > > > >
> > > > > About the code: I have not thought about putting it to
> > contrib-package,
> > > > but
> > > > > I can. What do you think is suitable for flink-contrib?
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Felix
> > > > >
> > > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
> > <javascript:;>>:
> > > > >
> > > > > > Wow, very nice work!
> > > > > >
> > > > > > It looks impressive at first sight, no comments :-)
> > > > > >
> > > > > > Just one question: Do you want this to be a standalone tutorial,
> or
> > > are
> > > > > you
> > > > > > thinking of putting this into the Flink wiki, or some code even
> > into
> > > > > > "flink-contrib" ?
> > > > > >
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> > > [hidden email] <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everybody,
> > > > > > >
> > > > > > > I am working currently on a tutorial/article about how/when/why
> > to
> > > > use
> > > > > > > Parquet on Flink.
> > > > > > >
> > > > > > > You can find the pdf version here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > > > > >
> > > > > > > The git repository with all the code examples can be found
> here:
> > > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > > > > >
> > > > > > > What do you think about it? I am happy about every feedback I
> can
> > > get
> > > > > :)
> > > > > > >
> > > > > > > Have a nice Sunday,
> > > > > > >
> > > > > > > Felix
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Flavio Pompermaier
I was looking at this great example and I'd like to ask you which
serialization framework is the best if I have to serialize
Tuple3<String,String,byte[]> with Parquet.
The syntax I like the most is the Thrift one but I can't see all the pros
and cons of using it and I'd like to hear your opinion here.

Thanks in advance,
Flavio

On Fri, Apr 10, 2015 at 2:52 PM, Robert Metzger <[hidden email]> wrote:

> +1 for a blog post
>
> On Tue, Apr 7, 2015 at 5:17 PM, Henry Saputra <[hidden email]>
> wrote:
>
> > +1 to the idea.
> >
> > Awesome work, Felix
> >
> > On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]> wrote:
> >
> > > Hi Felix,
> > >
> > > Very nice informative read.
> > >
> > > +1 for a short blog post and a full version in the wiki.
> > > +1 for putting this into flink-contrib
> > >
> > >
> > > On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]
> > > <javascript:;>> wrote:
> > >
> > > > Very nice article!
> > > > How about adding the full article to the wiki and having a shorter
> > > version
> > > > as a blog post (with a link to the wiki)?
> > > > Adding the code to contrib would be great!
> > > >
> > > > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]
> > > <javascript:;>>:
> > > >
> > > > > Looks very nice! Would love to see a blog post on that!
> > > > >
> > > > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <
> > [hidden email]
> > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > The intention was to post it on the blog, but if you think it
> would
> > > > > better
> > > > > > fit into the wiki, that would be also fine :)
> > > > > >
> > > > > > About the code: I have not thought about putting it to
> > > contrib-package,
> > > > > but
> > > > > > I can. What do you think is suitable for flink-contrib?
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Felix
> > > > > >
> > > > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
> > > <javascript:;>>:
> > > > > >
> > > > > > > Wow, very nice work!
> > > > > > >
> > > > > > > It looks impressive at first sight, no comments :-)
> > > > > > >
> > > > > > > Just one question: Do you want this to be a standalone
> tutorial,
> > or
> > > > are
> > > > > > you
> > > > > > > thinking of putting this into the Flink wiki, or some code even
> > > into
> > > > > > > "flink-contrib" ?
> > > > > > >
> > > > > > > Stephan
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> > > > [hidden email] <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everybody,
> > > > > > > >
> > > > > > > > I am working currently on a tutorial/article about
> how/when/why
> > > to
> > > > > use
> > > > > > > > Parquet on Flink.
> > > > > > > >
> > > > > > > > You can find the pdf version here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > > > > > >
> > > > > > > > The git repository with all the code examples can be found
> > here:
> > > > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > > > > > >
> > > > > > > > What do you think about it? I am happy about every feedback I
> > can
> > > > get
> > > > > > :)
> > > > > > > >
> > > > > > > > Have a nice Sunday,
> > > > > > > >
> > > > > > > > Felix
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Felix Neutatz
Hi Flavio,

in Thrift you can try:

struct FlavioTuple {
    1: optional string f1;
    2: optional string f2;
    3: optional list<byte> f3;
}

See: http://diwakergupta.github.io/thrift-missing-guide/

I like Thrift the most, because the API for Thrift in Parquet is the
easiest.

Have fun with Parquet :)

Best regards,

Felix

2015-04-24 12:28 GMT+02:00 Flavio Pompermaier <[hidden email]>:

> I was looking at this great example and I'd like to ask you which
> serialization framework is the best if I have to serialize
> Tuple3<String,String,byte[]> with Parquet.
> The syntax I like the most is the Thrift one but I can't see all the pros
> and cons of using it and I'd like to hear your opinion here.
>
> Thanks in advance,
> Flavio
>
> On Fri, Apr 10, 2015 at 2:52 PM, Robert Metzger <[hidden email]>
> wrote:
>
> > +1 for a blog post
> >
> > On Tue, Apr 7, 2015 at 5:17 PM, Henry Saputra <[hidden email]>
> > wrote:
> >
> > > +1 to the idea.
> > >
> > > Awesome work, Felix
> > >
> > > On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]> wrote:
> > >
> > > > Hi Felix,
> > > >
> > > > Very nice informative read.
> > > >
> > > > +1 for a short blog post and a full version in the wiki.
> > > > +1 for putting this into flink-contrib
> > > >
> > > >
> > > > On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]
> > > > <javascript:;>> wrote:
> > > >
> > > > > Very nice article!
> > > > > How about adding the full article to the wiki and having a shorter
> > > > version
> > > > > as a blog post (with a link to the wiki)?
> > > > > Adding the code to contrib would be great!
> > > > >
> > > > > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]
> > > > <javascript:;>>:
> > > > >
> > > > > > Looks very nice! Would love to see a blog post on that!
> > > > > >
> > > > > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <
> > > [hidden email]
> > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > The intention was to post it on the blog, but if you think it
> > would
> > > > > > better
> > > > > > > fit into the wiki, that would be also fine :)
> > > > > > >
> > > > > > > About the code: I have not thought about putting it to
> > > > contrib-package,
> > > > > > but
> > > > > > > I can. What do you think is suitable for flink-contrib?
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Felix
> > > > > > >
> > > > > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
> > > > <javascript:;>>:
> > > > > > >
> > > > > > > > Wow, very nice work!
> > > > > > > >
> > > > > > > > It looks impressive at first sight, no comments :-)
> > > > > > > >
> > > > > > > > Just one question: Do you want this to be a standalone
> > tutorial,
> > > or
> > > > > are
> > > > > > > you
> > > > > > > > thinking of putting this into the Flink wiki, or some code
> even
> > > > into
> > > > > > > > "flink-contrib" ?
> > > > > > > >
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> > > > > [hidden email] <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everybody,
> > > > > > > > >
> > > > > > > > > I am working currently on a tutorial/article about
> > how/when/why
> > > > to
> > > > > > use
> > > > > > > > > Parquet on Flink.
> > > > > > > > >
> > > > > > > > > You can find the pdf version here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > > > > > > >
> > > > > > > > > The git repository with all the code examples can be found
> > > here:
> > > > > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > > > > > > >
> > > > > > > > > What do you think about it? I am happy about every
> feedback I
> > > can
> > > > > get
> > > > > > > :)
> > > > > > > > >
> > > > > > > > > Have a nice Sunday,
> > > > > > > > >
> > > > > > > > > Felix
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Flavio Pompermaier
Thanks Felix,
Thanks fir the response!
I'm looking forward to use it!
On Apr 24, 2015 9:01 PM, "Felix Neutatz" <[hidden email]> wrote:

> Hi Flavio,
>
> in Thrift you can try:
>
> struct FlavioTuple {
>     1: optional string f1;
>     2: optional string f2;
>     3: optional list<byte> f3;
> }
>
> See: http://diwakergupta.github.io/thrift-missing-guide/
>
> I like Thrift the most, because the API for Thrift in Parquet is the
> easiest.
>
> Have fun with Parquet :)
>
> Best regards,
>
> Felix
>
> 2015-04-24 12:28 GMT+02:00 Flavio Pompermaier <[hidden email]>:
>
> > I was looking at this great example and I'd like to ask you which
> > serialization framework is the best if I have to serialize
> > Tuple3<String,String,byte[]> with Parquet.
> > The syntax I like the most is the Thrift one but I can't see all the pros
> > and cons of using it and I'd like to hear your opinion here.
> >
> > Thanks in advance,
> > Flavio
> >
> > On Fri, Apr 10, 2015 at 2:52 PM, Robert Metzger <[hidden email]>
> > wrote:
> >
> > > +1 for a blog post
> > >
> > > On Tue, Apr 7, 2015 at 5:17 PM, Henry Saputra <[hidden email]
> >
> > > wrote:
> > >
> > > > +1 to the idea.
> > > >
> > > > Awesome work, Felix
> > > >
> > > > On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]>
> wrote:
> > > >
> > > > > Hi Felix,
> > > > >
> > > > > Very nice informative read.
> > > > >
> > > > > +1 for a short blog post and a full version in the wiki.
> > > > > +1 for putting this into flink-contrib
> > > > >
> > > > >
> > > > > On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > > Very nice article!
> > > > > > How about adding the full article to the wiki and having a
> shorter
> > > > > version
> > > > > > as a blog post (with a link to the wiki)?
> > > > > > Adding the code to contrib would be great!
> > > > > >
> > > > > > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]
> > > > > <javascript:;>>:
> > > > > >
> > > > > > > Looks very nice! Would love to see a blog post on that!
> > > > > > >
> > > > > > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <
> > > > [hidden email]
> > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > The intention was to post it on the blog, but if you think it
> > > would
> > > > > > > better
> > > > > > > > fit into the wiki, that would be also fine :)
> > > > > > > >
> > > > > > > > About the code: I have not thought about putting it to
> > > > > contrib-package,
> > > > > > > but
> > > > > > > > I can. What do you think is suitable for flink-contrib?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Felix
> > > > > > > >
> > > > > > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
> > > > > <javascript:;>>:
> > > > > > > >
> > > > > > > > > Wow, very nice work!
> > > > > > > > >
> > > > > > > > > It looks impressive at first sight, no comments :-)
> > > > > > > > >
> > > > > > > > > Just one question: Do you want this to be a standalone
> > > tutorial,
> > > > or
> > > > > > are
> > > > > > > > you
> > > > > > > > > thinking of putting this into the Flink wiki, or some code
> > even
> > > > > into
> > > > > > > > > "flink-contrib" ?
> > > > > > > > >
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> > > > > > [hidden email] <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everybody,
> > > > > > > > > >
> > > > > > > > > > I am working currently on a tutorial/article about
> > > how/when/why
> > > > > to
> > > > > > > use
> > > > > > > > > > Parquet on Flink.
> > > > > > > > > >
> > > > > > > > > > You can find the pdf version here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> > > > > > > > > >
> > > > > > > > > > The git repository with all the code examples can be
> found
> > > > here:
> > > > > > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> > > > > > > > > >
> > > > > > > > > > What do you think about it? I am happy about every
> > feedback I
> > > > can
> > > > > > get
> > > > > > > > :)
> > > > > > > > > >
> > > > > > > > > > Have a nice Sunday,
> > > > > > > > > >
> > > > > > > > > > Felix
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Flavio Pompermaier
Hi Felix,
I was looking at your code and I don't see any parquet.block.size settings.
Do you think it is safe to keep defaults?

On Fri, Apr 24, 2015 at 11:19 PM, Flavio Pompermaier <[hidden email]>
wrote:

> Thanks Felix,
> Thanks fir the response!
> I'm looking forward to use it!
> On Apr 24, 2015 9:01 PM, "Felix Neutatz" <[hidden email]> wrote:
>
>> Hi Flavio,
>>
>> in Thrift you can try:
>>
>> struct FlavioTuple {
>>     1: optional string f1;
>>     2: optional string f2;
>>     3: optional list<byte> f3;
>> }
>>
>> See: http://diwakergupta.github.io/thrift-missing-guide/
>>
>> I like Thrift the most, because the API for Thrift in Parquet is the
>> easiest.
>>
>> Have fun with Parquet :)
>>
>> Best regards,
>>
>> Felix
>>
>> 2015-04-24 12:28 GMT+02:00 Flavio Pompermaier <[hidden email]>:
>>
>> > I was looking at this great example and I'd like to ask you which
>> > serialization framework is the best if I have to serialize
>> > Tuple3<String,String,byte[]> with Parquet.
>> > The syntax I like the most is the Thrift one but I can't see all the
>> pros
>> > and cons of using it and I'd like to hear your opinion here.
>> >
>> > Thanks in advance,
>> > Flavio
>> >
>> > On Fri, Apr 10, 2015 at 2:52 PM, Robert Metzger <[hidden email]>
>> > wrote:
>> >
>> > > +1 for a blog post
>> > >
>> > > On Tue, Apr 7, 2015 at 5:17 PM, Henry Saputra <
>> [hidden email]>
>> > > wrote:
>> > >
>> > > > +1 to the idea.
>> > > >
>> > > > Awesome work, Felix
>> > > >
>> > > > On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]>
>> wrote:
>> > > >
>> > > > > Hi Felix,
>> > > > >
>> > > > > Very nice informative read.
>> > > > >
>> > > > > +1 for a short blog post and a full version in the wiki.
>> > > > > +1 for putting this into flink-contrib
>> > > > >
>> > > > >
>> > > > > On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <[hidden email]
>> > > > > <javascript:;>> wrote:
>> > > > >
>> > > > > > Very nice article!
>> > > > > > How about adding the full article to the wiki and having a
>> shorter
>> > > > > version
>> > > > > > as a blog post (with a link to the wiki)?
>> > > > > > Adding the code to contrib would be great!
>> > > > > >
>> > > > > > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <[hidden email]
>> > > > > <javascript:;>>:
>> > > > > >
>> > > > > > > Looks very nice! Would love to see a blog post on that!
>> > > > > > >
>> > > > > > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <
>> > > > [hidden email]
>> > > > > <javascript:;>>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > The intention was to post it on the blog, but if you think
>> it
>> > > would
>> > > > > > > better
>> > > > > > > > fit into the wiki, that would be also fine :)
>> > > > > > > >
>> > > > > > > > About the code: I have not thought about putting it to
>> > > > > contrib-package,
>> > > > > > > but
>> > > > > > > > I can. What do you think is suitable for flink-contrib?
>> > > > > > > >
>> > > > > > > > Best regards,
>> > > > > > > >
>> > > > > > > > Felix
>> > > > > > > >
>> > > > > > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
>> > > > > <javascript:;>>:
>> > > > > > > >
>> > > > > > > > > Wow, very nice work!
>> > > > > > > > >
>> > > > > > > > > It looks impressive at first sight, no comments :-)
>> > > > > > > > >
>> > > > > > > > > Just one question: Do you want this to be a standalone
>> > > tutorial,
>> > > > or
>> > > > > > are
>> > > > > > > > you
>> > > > > > > > > thinking of putting this into the Flink wiki, or some code
>> > even
>> > > > > into
>> > > > > > > > > "flink-contrib" ?
>> > > > > > > > >
>> > > > > > > > > Stephan
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
>> > > > > > [hidden email] <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi everybody,
>> > > > > > > > > >
>> > > > > > > > > > I am working currently on a tutorial/article about
>> > > how/when/why
>> > > > > to
>> > > > > > > use
>> > > > > > > > > > Parquet on Flink.
>> > > > > > > > > >
>> > > > > > > > > > You can find the pdf version here:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
>> > > > > > > > > >
>> > > > > > > > > > The git repository with all the code examples can be
>> found
>> > > > here:
>> > > > > > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
>> > > > > > > > > >
>> > > > > > > > > > What do you think about it? I am happy about every
>> > feedback I
>> > > > can
>> > > > > > get
>> > > > > > > > :)
>> > > > > > > > > >
>> > > > > > > > > > Have a nice Sunday,
>> > > > > > > > > >
>> > > > > > > > > > Felix
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Parquet Article / Tutorial

Felix Neutatz
Hi Flavio,

changing the block size may help. But I haven't played around with it yet.

Best regards,
Felix
Am 11.05.2015 12:23 nachm. schrieb "Flavio Pompermaier" <
[hidden email]>:

> Hi Felix,
> I was looking at your code and I don't see any parquet.block.size settings.
> Do you think it is safe to keep defaults?
>
> On Fri, Apr 24, 2015 at 11:19 PM, Flavio Pompermaier <[hidden email]
> >
> wrote:
>
> > Thanks Felix,
> > Thanks fir the response!
> > I'm looking forward to use it!
> > On Apr 24, 2015 9:01 PM, "Felix Neutatz" <[hidden email]> wrote:
> >
> >> Hi Flavio,
> >>
> >> in Thrift you can try:
> >>
> >> struct FlavioTuple {
> >>     1: optional string f1;
> >>     2: optional string f2;
> >>     3: optional list<byte> f3;
> >> }
> >>
> >> See: http://diwakergupta.github.io/thrift-missing-guide/
> >>
> >> I like Thrift the most, because the API for Thrift in Parquet is the
> >> easiest.
> >>
> >> Have fun with Parquet :)
> >>
> >> Best regards,
> >>
> >> Felix
> >>
> >> 2015-04-24 12:28 GMT+02:00 Flavio Pompermaier <[hidden email]>:
> >>
> >> > I was looking at this great example and I'd like to ask you which
> >> > serialization framework is the best if I have to serialize
> >> > Tuple3<String,String,byte[]> with Parquet.
> >> > The syntax I like the most is the Thrift one but I can't see all the
> >> pros
> >> > and cons of using it and I'd like to hear your opinion here.
> >> >
> >> > Thanks in advance,
> >> > Flavio
> >> >
> >> > On Fri, Apr 10, 2015 at 2:52 PM, Robert Metzger <[hidden email]>
> >> > wrote:
> >> >
> >> > > +1 for a blog post
> >> > >
> >> > > On Tue, Apr 7, 2015 at 5:17 PM, Henry Saputra <
> >> [hidden email]>
> >> > > wrote:
> >> > >
> >> > > > +1 to the idea.
> >> > > >
> >> > > > Awesome work, Felix
> >> > > >
> >> > > > On Tuesday, April 7, 2015, Maximilian Michels <[hidden email]>
> >> wrote:
> >> > > >
> >> > > > > Hi Felix,
> >> > > > >
> >> > > > > Very nice informative read.
> >> > > > >
> >> > > > > +1 for a short blog post and a full version in the wiki.
> >> > > > > +1 for putting this into flink-contrib
> >> > > > >
> >> > > > >
> >> > > > > On Tue, Apr 7, 2015 at 1:46 PM, Fabian Hueske <
> [hidden email]
> >> > > > > <javascript:;>> wrote:
> >> > > > >
> >> > > > > > Very nice article!
> >> > > > > > How about adding the full article to the wiki and having a
> >> shorter
> >> > > > > version
> >> > > > > > as a blog post (with a link to the wiki)?
> >> > > > > > Adding the code to contrib would be great!
> >> > > > > >
> >> > > > > > 2015-04-07 12:45 GMT+02:00 Kostas Tzoumas <
> [hidden email]
> >> > > > > <javascript:;>>:
> >> > > > > >
> >> > > > > > > Looks very nice! Would love to see a blog post on that!
> >> > > > > > >
> >> > > > > > > On Mon, Apr 6, 2015 at 7:19 PM, Felix Neutatz <
> >> > > > [hidden email]
> >> > > > > <javascript:;>>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > The intention was to post it on the blog, but if you think
> >> it
> >> > > would
> >> > > > > > > better
> >> > > > > > > > fit into the wiki, that would be also fine :)
> >> > > > > > > >
> >> > > > > > > > About the code: I have not thought about putting it to
> >> > > > > contrib-package,
> >> > > > > > > but
> >> > > > > > > > I can. What do you think is suitable for flink-contrib?
> >> > > > > > > >
> >> > > > > > > > Best regards,
> >> > > > > > > >
> >> > > > > > > > Felix
> >> > > > > > > >
> >> > > > > > > > 2015-04-06 14:57 GMT+02:00 Stephan Ewen <[hidden email]
> >> > > > > <javascript:;>>:
> >> > > > > > > >
> >> > > > > > > > > Wow, very nice work!
> >> > > > > > > > >
> >> > > > > > > > > It looks impressive at first sight, no comments :-)
> >> > > > > > > > >
> >> > > > > > > > > Just one question: Do you want this to be a standalone
> >> > > tutorial,
> >> > > > or
> >> > > > > > are
> >> > > > > > > > you
> >> > > > > > > > > thinking of putting this into the Flink wiki, or some
> code
> >> > even
> >> > > > > into
> >> > > > > > > > > "flink-contrib" ?
> >> > > > > > > > >
> >> > > > > > > > > Stephan
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Sun, Apr 5, 2015 at 3:30 PM, Felix Neutatz <
> >> > > > > > [hidden email] <javascript:;>>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi everybody,
> >> > > > > > > > > >
> >> > > > > > > > > > I am working currently on a tutorial/article about
> >> > > how/when/why
> >> > > > > to
> >> > > > > > > use
> >> > > > > > > > > > Parquet on Flink.
> >> > > > > > > > > >
> >> > > > > > > > > > You can find the pdf version here:
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/FelixNeutatz/parquet-flinktacular/blob/master/tutorial/parquet_flinktacular.pdf
> >> > > > > > > > > >
> >> > > > > > > > > > The git repository with all the code examples can be
> >> found
> >> > > > here:
> >> > > > > > > > > > https://github.com/FelixNeutatz/parquet-flinktacular/
> >> > > > > > > > > >
> >> > > > > > > > > > What do you think about it? I am happy about every
> >> > feedback I
> >> > > > can
> >> > > > > > get
> >> > > > > > > > :)
> >> > > > > > > > > >
> >> > > > > > > > > > Have a nice Sunday,
> >> > > > > > > > > >
> >> > > > > > > > > > Felix
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>