(DEPRECATED) Apache Flink Mailing List archive.

[Parquet support]

Classic

List

Threaded

16 messages Options

Etienne Chauchot

[Parquet support]

Hi all,

I've been playing with Parquet with SQL and Avro lately. I've found some
bugs:

1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
submitted a PR on this one (https://github.com/apache/flink/pull/14961)

2. https://issues.apache.org/jira/browse/FLINK-21389

3. https://issues.apache.org/jira/browse/FLINK-21468

I've already started to work on this ticket:
https://issues.apache.org/jira/browse/FLINK-21393

I'd be happy to receive your comments on these tickets

Best

Etienne Chauchot

Jingsong Li

Re: [Parquet support]

Hi Etienne,

Thanks for your reporting.

There are indeed many problems. There is no doubt that we need to improve
our current format implementation.

But ParquetTableSource and ParquetInputFormat are legacy implementations
with legacy interfaces. We have introduced new interfaces for execution and
SQL. You can see:
- ParquetColumnarRowInputFormat with BulkFormat interface. It is just for
columnar row reading, not support complex types, we need
migrate ParquetInputFormat to the new BulkFormat interface.
- FileSystemTableSource with DynamicTableSource interface, It is a generic
FileSystem source for all formats, we can just use it for parquet too.

Considering ParquetTableSource and ParquetInputFormat are legacy
interfaces, I think we can finish migration work first, what do you think?

Best,
Jingsong

On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]>
wrote:

> Hi all,
>
> I've been playing with Parquet with SQL and Avro lately. I've found some
> bugs:
>
> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
> submitted a PR on this one (https://github.com/apache/flink/pull/14961)
>
> 2. https://issues.apache.org/jira/browse/FLINK-21389
>
> 3. https://issues.apache.org/jira/browse/FLINK-21468
>
> I've already started to work on this ticket:
> https://issues.apache.org/jira/browse/FLINK-21393
>
>
> I'd be happy to receive your comments on these tickets
>
>
> Best
>
> Etienne Chauchot
>
>
>

--
Best, Jingsong Lee

Peter Huang

Re: [Parquet support]

Hi Jingsong,

Thanks for pointing this out. Actually, I planned to work on changing
interfaces ParquetTableSource and ParquetInputFormat.
After refactoring the code, I may also help to fix the issue in
https://issues.apache.org/jira/browse/FLINK-21468.

Best Regards
Peter Huang

On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li <[hidden email]> wrote:

> Hi Etienne,
>
> Thanks for your reporting.
>
> There are indeed many problems. There is no doubt that we need to improve
> our current format implementation.
>
> But ParquetTableSource and ParquetInputFormat are legacy implementations
> with legacy interfaces. We have introduced new interfaces for execution and
> SQL. You can see:
> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just for
> columnar row reading, not support complex types, we need
> migrate ParquetInputFormat to the new BulkFormat interface.
> - FileSystemTableSource with DynamicTableSource interface, It is a generic
> FileSystem source for all formats, we can just use it for parquet too.
>
> Considering ParquetTableSource and ParquetInputFormat are legacy
> interfaces, I think we can finish migration work first, what do you think?
>
> Best,
> Jingsong
>
> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]>
> wrote:
>
> > Hi all,
> >
> > I've been playing with Parquet with SQL and Avro lately. I've found some
> > bugs:
> >
> > 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
> > submitted a PR on this one (https://github.com/apache/flink/pull/14961)
> >
> > 2. https://issues.apache.org/jira/browse/FLINK-21389
> >
> > 3. https://issues.apache.org/jira/browse/FLINK-21468
> >
> > I've already started to work on this ticket:
> > https://issues.apache.org/jira/browse/FLINK-21393
> >
> >
> > I'd be happy to receive your comments on these tickets
> >
> >
> > Best
> >
> > Etienne Chauchot
> >
> >
> >
>
> --
> Best, Jingsong Lee
>

Etienne Chauchot

Re: [Parquet support]

Hi,

Thanks guys for the comments !

I did not know it was legacy. I will give the new sources a try.

Jingsong, when you say "migrate ParquetInputFormat to the new BulkFormat
interface", do you mean that the new ParquetColumnarRowInputFormat is
not fully functional yet?

In the meantime, if you agree, I think I'm still gonna submit a PR for
https://issues.apache.org/jira/browse/FLINK-21393 because I need it on
an urgent task I'm doing.

Best

Etienne

On 24/02/2021 03:41, Peter Huang wrote:

> Hi Jingsong,
>
> Thanks for pointing this out. Actually, I planned to work on changing
> interfaces ParquetTableSource and ParquetInputFormat.
> After refactoring the code, I may also help to fix the issue in
> https://issues.apache.org/jira/browse/FLINK-21468.
>
> Best Regards
> Peter Huang
>
> On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li <[hidden email]> wrote:
>
>> Hi Etienne,
>>
>> Thanks for your reporting.
>>
>> There are indeed many problems. There is no doubt that we need to improve
>> our current format implementation.
>>
>> But ParquetTableSource and ParquetInputFormat are legacy implementations
>> with legacy interfaces. We have introduced new interfaces for execution and
>> SQL. You can see:
>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just for
>> columnar row reading, not support complex types, we need
>> migrate ParquetInputFormat to the new BulkFormat interface.
>> - FileSystemTableSource with DynamicTableSource interface, It is a generic
>> FileSystem source for all formats, we can just use it for parquet too.
>>
>> Considering ParquetTableSource and ParquetInputFormat are legacy
>> interfaces, I think we can finish migration work first, what do you think?
>>
>> Best,
>> Jingsong
>>
>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]>
>> wrote:
>>
>>> Hi all,
>>>
>>> I've been playing with Parquet with SQL and Avro lately. I've found some
>>> bugs:
>>>
>>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>> submitted a PR on this one (https://github.com/apache/flink/pull/14961)
>>>
>>> 2. https://issues.apache.org/jira/browse/FLINK-21389
>>>
>>> 3. https://issues.apache.org/jira/browse/FLINK-21468
>>>
>>> I've already started to work on this ticket:
>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>
>>>
>>> I'd be happy to receive your comments on these tickets
>>>
>>>
>>> Best
>>>
>>> Etienne Chauchot
>>>
>>>
>>>
>> --
>> Best, Jingsong Lee
>>

Jingsong Li

Re: [Parquet support]

Hi Etienne,

ParquetColumnarRowInputFormat is not fully functional yet, it has a good
performance, but it is hard to support complex types, like array and map...
So I think a migrated ParquetInputFormat version is required.

Best,
Jingsong

On Wed, Feb 24, 2021 at 3:43 PM Etienne Chauchot <[hidden email]>
wrote:

> Hi,
>
> Thanks guys for the comments !
>
> I did not know it was legacy. I will give the new sources a try.
>
> Jingsong, when you say "migrate ParquetInputFormat to the new BulkFormat
> interface", do you mean that the new ParquetColumnarRowInputFormat is
> not fully functional yet?
>
> In the meantime, if you agree, I think I'm still gonna submit a PR for
> https://issues.apache.org/jira/browse/FLINK-21393 because I need it on
> an urgent task I'm doing.
>
> Best
>
> Etienne
>
> On 24/02/2021 03:41, Peter Huang wrote:
> > Hi Jingsong,
> >
> > Thanks for pointing this out. Actually, I planned to work on changing
> > interfaces ParquetTableSource and ParquetInputFormat.
> > After refactoring the code, I may also help to fix the issue in
> > https://issues.apache.org/jira/browse/FLINK-21468.
> >
> > Best Regards
> > Peter Huang
> >
> > On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li <[hidden email]>
> wrote:
> >
> >> Hi Etienne,
> >>
> >> Thanks for your reporting.
> >>
> >> There are indeed many problems. There is no doubt that we need to
> improve
> >> our current format implementation.
> >>
> >> But ParquetTableSource and ParquetInputFormat are legacy implementations
> >> with legacy interfaces. We have introduced new interfaces for execution
> and
> >> SQL. You can see:
> >> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
> for
> >> columnar row reading, not support complex types, we need
> >> migrate ParquetInputFormat to the new BulkFormat interface.
> >> - FileSystemTableSource with DynamicTableSource interface, It is a
> generic
> >> FileSystem source for all formats, we can just use it for parquet too.
> >>
> >> Considering ParquetTableSource and ParquetInputFormat are legacy
> >> interfaces, I think we can finish migration work first, what do you
> think?
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]
> >
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I've been playing with Parquet with SQL and Avro lately. I've found
> some
> >>> bugs:
> >>>
> >>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
> >>> submitted a PR on this one (https://github.com/apache/flink/pull/14961
> )
> >>>
> >>> 2. https://issues.apache.org/jira/browse/FLINK-21389
> >>>
> >>> 3. https://issues.apache.org/jira/browse/FLINK-21468
> >>>
> >>> I've already started to work on this ticket:
> >>> https://issues.apache.org/jira/browse/FLINK-21393
> >>>
> >>>
> >>> I'd be happy to receive your comments on these tickets
> >>>
> >>>
> >>> Best
> >>>
> >>> Etienne Chauchot
> >>>
> >>>
> >>>
> >> --
> >> Best, Jingsong Lee
> >>
>

--
Best, Jingsong Lee

Etienne Chauchot

Re: [Parquet support]

Hi all,

Jingsong, thanks it makes sense.

Besides, sorry but I found another bug in ParquetInputFormat:

https://issues.apache.org/jira/browse/FLINK-21520

For my urgent needs, I'll workaround by filtering the dataSet rather
than applying the filter in the ParquetInputFormat at source reading time.

Etienne Chauchot

On 25/02/2021 08:34, Jingsong Li wrote:

> Hi Etienne,
>
> ParquetColumnarRowInputFormat is not fully functional yet, it has a good
> performance, but it is hard to support complex types, like array and map...
> So I think a migrated ParquetInputFormat version is required.
>
> Best,
> Jingsong
>
> On Wed, Feb 24, 2021 at 3:43 PM Etienne Chauchot <[hidden email]>
> wrote:
>
>> Hi,
>>
>> Thanks guys for the comments !
>>
>> I did not know it was legacy. I will give the new sources a try.
>>
>> Jingsong, when you say "migrate ParquetInputFormat to the new BulkFormat
>> interface", do you mean that the new ParquetColumnarRowInputFormat is
>> not fully functional yet?
>>
>> In the meantime, if you agree, I think I'm still gonna submit a PR for
>> https://issues.apache.org/jira/browse/FLINK-21393 because I need it on
>> an urgent task I'm doing.
>>
>> Best
>>
>> Etienne
>>
>> On 24/02/2021 03:41, Peter Huang wrote:
>>> Hi Jingsong,
>>>
>>> Thanks for pointing this out. Actually, I planned to work on changing
>>> interfaces ParquetTableSource and ParquetInputFormat.
>>> After refactoring the code, I may also help to fix the issue in
>>> https://issues.apache.org/jira/browse/FLINK-21468.
>>>
>>> Best Regards
>>> Peter Huang
>>>
>>> On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li <[hidden email]>
>> wrote:
>>>> Hi Etienne,
>>>>
>>>> Thanks for your reporting.
>>>>
>>>> There are indeed many problems. There is no doubt that we need to
>> improve
>>>> our current format implementation.
>>>>
>>>> But ParquetTableSource and ParquetInputFormat are legacy implementations
>>>> with legacy interfaces. We have introduced new interfaces for execution
>> and
>>>> SQL. You can see:
>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
>> for
>>>> columnar row reading, not support complex types, we need
>>>> migrate ParquetInputFormat to the new BulkFormat interface.
>>>> - FileSystemTableSource with DynamicTableSource interface, It is a
>> generic
>>>> FileSystem source for all formats, we can just use it for parquet too.
>>>>
>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
>>>> interfaces, I think we can finish migration work first, what do you
>> think?
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I've been playing with Parquet with SQL and Avro lately. I've found
>> some
>>>>> bugs:
>>>>>
>>>>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>>>> submitted a PR on this one (https://github.com/apache/flink/pull/14961
>> )
>>>>> 2. https://issues.apache.org/jira/browse/FLINK-21389
>>>>>
>>>>> 3. https://issues.apache.org/jira/browse/FLINK-21468
>>>>>
>>>>> I've already started to work on this ticket:
>>>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>>>
>>>>>
>>>>> I'd be happy to receive your comments on these tickets
>>>>>
>>>>>
>>>>> Best
>>>>>
>>>>> Etienne Chauchot
>>>>>
>>>>>
>>>>>
>>>> --
>>>> Best, Jingsong Lee
>>>>
>

Etienne Chauchot

Re: [Parquet support]

In reply to this post by Jingsong Li

Hi all,

I just submitted another parquet PR that adds ParquetAvroInputFormat
(I'm using it in a benchmark I'm coding). If anyone is interested in
reviewing it, be my guest:

https://github.com/apache/flink/pull/15156

I have also an older parquet PR that fixes a format conversion bug that
is waiting for merge if anyone can review it also (already 1 approval of
a non-committer, thanks @HuangZhenQiu <https://github.com/HuangZhenQiu>):

https://github.com/apache/flink/pull/14961

If I have time, I'll also tackle the other parquet tickets that I opened
lately

Best

Etienne

On 25/02/2021 08:34, Jingsong Li wrote:

Etienne Chauchot

Re: [Parquet support]

Hi,

I forgot to mention that I submitted the new ParquetAvroInputFormat to
master (1.13) but it is made to work for 1.12.x (last release) also and
I'm using it with Flink 1.12.x.

Maybe it could be a good candidate to be included in an upcoming 1.12.3
release, WDYT ?

Best

Etienne

On 11/03/2021 17:17, Etienne Chauchot wrote:

>
> Hi all,
>
> I just submitted another parquet PR that adds ParquetAvroInputFormat
> (I'm using it in a benchmark I'm coding). If anyone is interested in
> reviewing it, be my guest:
>
> https://github.com/apache/flink/pull/15156
>
> I have also an older parquet PR that fixes a format conversion bug
> that is waiting for merge if anyone can review it also (already 1
> approval of a non-committer, thanks @HuangZhenQiu
> <https://github.com/HuangZhenQiu>):
>
> https://github.com/apache/flink/pull/14961
>
> If I have time, I'll also tackle the other parquet tickets that I
> opened lately
>
> Best
>
> Etienne
>
> On 25/02/2021 08:34, Jingsong Li wrote:
>> Hi Etienne,
>>
>> ParquetColumnarRowInputFormat is not fully functional yet, it has a good
>> performance, but it is hard to support complex types, like array and map...
>> So I think a migrated ParquetInputFormat version is required.
>>
>> Best,
>> Jingsong
>>
>> On Wed, Feb 24, 2021 at 3:43 PM Etienne Chauchot<[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks guys for the comments !
>>>
>>> I did not know it was legacy. I will give the new sources a try.
>>>
>>> Jingsong, when you say "migrate ParquetInputFormat to the new BulkFormat
>>> interface", do you mean that the new ParquetColumnarRowInputFormat is
>>> not fully functional yet?
>>>
>>> In the meantime, if you agree, I think I'm still gonna submit a PR for
>>> https://issues.apache.org/jira/browse/FLINK-21393 because I need it on
>>> an urgent task I'm doing.
>>>
>>> Best
>>>
>>> Etienne
>>>
>>> On 24/02/2021 03:41, Peter Huang wrote:
>>>> Hi Jingsong,
>>>>
>>>> Thanks for pointing this out. Actually, I planned to work on changing
>>>> interfaces ParquetTableSource and ParquetInputFormat.
>>>> After refactoring the code, I may also help to fix the issue in
>>>> https://issues.apache.org/jira/browse/FLINK-21468.
>>>>
>>>> Best Regards
>>>> Peter Huang
>>>>
>>>> On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li<[hidden email]>
>>> wrote:
>>>>> Hi Etienne,
>>>>>
>>>>> Thanks for your reporting.
>>>>>
>>>>> There are indeed many problems. There is no doubt that we need to
>>> improve
>>>>> our current format implementation.
>>>>>
>>>>> But ParquetTableSource and ParquetInputFormat are legacy implementations
>>>>> with legacy interfaces. We have introduced new interfaces for execution
>>> and
>>>>> SQL. You can see:
>>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
>>> for
>>>>> columnar row reading, not support complex types, we need
>>>>> migrate ParquetInputFormat to the new BulkFormat interface.
>>>>> - FileSystemTableSource with DynamicTableSource interface, It is a
>>> generic
>>>>> FileSystem source for all formats, we can just use it for parquet too.
>>>>>
>>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
>>>>> interfaces, I think we can finish migration work first, what do you
>>> think?
>>>>> Best,
>>>>> Jingsong
>>>>>
>>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I've been playing with Parquet with SQL and Avro lately. I've found
>>> some
>>>>>> bugs:
>>>>>>
>>>>>> 1.https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>>>>> submitted a PR on this one (https://github.com/apache/flink/pull/14961
>>> )
>>>>>> 2.https://issues.apache.org/jira/browse/FLINK-21389
>>>>>>
>>>>>> 3.https://issues.apache.org/jira/browse/FLINK-21468
>>>>>>
>>>>>> I've already started to work on this ticket:
>>>>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>>>>
>>>>>>
>>>>>> I'd be happy to receive your comments on these tickets
>>>>>>
>>>>>>
>>>>>> Best
>>>>>>
>>>>>> Etienne Chauchot
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Best, Jingsong Lee
>>>>>

Jingsong Li

Re: [Parquet support]

Hi Etienne,

Sorry for the late reply,

I just merged your bug fixing.
I think you can submit a PR for release-1.12.

Best,
Jingsong

On Fri, Mar 12, 2021 at 12:22 AM Etienne Chauchot <[hidden email]>
wrote:

> Hi,
>
> I forgot to mention that I submitted the new ParquetAvroInputFormat to
> master (1.13) but it is made to work for 1.12.x (last release) also and
> I'm using it with Flink 1.12.x.
>
> Maybe it could be a good candidate to be included in an upcoming 1.12.3
> release, WDYT ?
>
> Best
>
> Etienne
>
> On 11/03/2021 17:17, Etienne Chauchot wrote:
> >
> > Hi all,
> >
> > I just submitted another parquet PR that adds ParquetAvroInputFormat
> > (I'm using it in a benchmark I'm coding). If anyone is interested in
> > reviewing it, be my guest:
> >
> > https://github.com/apache/flink/pull/15156
> >
> > I have also an older parquet PR that fixes a format conversion bug
> > that is waiting for merge if anyone can review it also (already 1
> > approval of a non-committer, thanks @HuangZhenQiu
> > <https://github.com/HuangZhenQiu>):
> >
> > https://github.com/apache/flink/pull/14961
> >
> > If I have time, I'll also tackle the other parquet tickets that I
> > opened lately
> >
> > Best
> >
> > Etienne
> >
> > On 25/02/2021 08:34, Jingsong Li wrote:
> >> Hi Etienne,
> >>
> >> ParquetColumnarRowInputFormat is not fully functional yet, it has a good
> >> performance, but it is hard to support complex types, like array and
> map...
> >> So I think a migrated ParquetInputFormat version is required.
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Wed, Feb 24, 2021 at 3:43 PM Etienne Chauchot<[hidden email]>
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks guys for the comments !
> >>>
> >>> I did not know it was legacy. I will give the new sources a try.
> >>>
> >>> Jingsong, when you say "migrate ParquetInputFormat to the new
> BulkFormat
> >>> interface", do you mean that the new ParquetColumnarRowInputFormat is
> >>> not fully functional yet?
> >>>
> >>> In the meantime, if you agree, I think I'm still gonna submit a PR for
> >>> https://issues.apache.org/jira/browse/FLINK-21393 because I need it
> on
> >>> an urgent task I'm doing.
> >>>
> >>> Best
> >>>
> >>> Etienne
> >>>
> >>> On 24/02/2021 03:41, Peter Huang wrote:
> >>>> Hi Jingsong,
> >>>>
> >>>> Thanks for pointing this out. Actually, I planned to work on changing
> >>>> interfaces ParquetTableSource and ParquetInputFormat.
> >>>> After refactoring the code, I may also help to fix the issue in
> >>>> https://issues.apache.org/jira/browse/FLINK-21468.
> >>>>
> >>>> Best Regards
> >>>> Peter Huang
> >>>>
> >>>> On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li<[hidden email]>
> >>> wrote:
> >>>>> Hi Etienne,
> >>>>>
> >>>>> Thanks for your reporting.
> >>>>>
> >>>>> There are indeed many problems. There is no doubt that we need to
> >>> improve
> >>>>> our current format implementation.
> >>>>>
> >>>>> But ParquetTableSource and ParquetInputFormat are legacy
> implementations
> >>>>> with legacy interfaces. We have introduced new interfaces for
> execution
> >>> and
> >>>>> SQL. You can see:
> >>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
> >>> for
> >>>>> columnar row reading, not support complex types, we need
> >>>>> migrate ParquetInputFormat to the new BulkFormat interface.
> >>>>> - FileSystemTableSource with DynamicTableSource interface, It is a
> >>> generic
> >>>>> FileSystem source for all formats, we can just use it for parquet
> too.
> >>>>>
> >>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
> >>>>> interfaces, I think we can finish migration work first, what do you
> >>> think?
> >>>>> Best,
> >>>>> Jingsong
> >>>>>
> >>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <
> [hidden email]
> >>>>> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I've been playing with Parquet with SQL and Avro lately. I've found
> >>> some
> >>>>>> bugs:
> >>>>>>
> >>>>>> 1.https://issues.apache.org/jira/browse/FLINK-21388 : I already
> >>>>>> submitted a PR on this one (
> https://github.com/apache/flink/pull/14961
> >>> )
> >>>>>> 2.https://issues.apache.org/jira/browse/FLINK-21389
> >>>>>>
> >>>>>> 3.https://issues.apache.org/jira/browse/FLINK-21468
> >>>>>>
> >>>>>> I've already started to work on this ticket:
> >>>>>> https://issues.apache.org/jira/browse/FLINK-21393
> >>>>>>
> >>>>>>
> >>>>>> I'd be happy to receive your comments on these tickets
> >>>>>>
> >>>>>>
> >>>>>> Best
> >>>>>>
> >>>>>> Etienne Chauchot
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> --
> >>>>> Best, Jingsong Lee
> >>>>>
>

--
Best, Jingsong Lee

Etienne Chauchot

Re: [Parquet support]

Hi Jingsong,

No problem for the delay. Thanks for merging the first parquet PR.

I'll submit the 2 PRs to 1.12 when they're all merged to master. For
that, I just have to submit a PR against this branch:
https://github.com/apache/flink/tree/release-1.12 ?

Best,

Etienne

On 12/03/2021 03:56, Jingsong Li wrote:

> Hi Etienne,
>
> Sorry for the late reply,
>
> I just merged your bug fixing.
> I think you can submit a PR for release-1.12.
>
> Best,
> Jingsong
>
> On Fri, Mar 12, 2021 at 12:22 AM Etienne Chauchot <[hidden email]>
> wrote:
>
>> Hi,
>>
>> I forgot to mention that I submitted the new ParquetAvroInputFormat to
>> master (1.13) but it is made to work for 1.12.x (last release) also and
>> I'm using it with Flink 1.12.x.
>>
>> Maybe it could be a good candidate to be included in an upcoming 1.12.3
>> release, WDYT ?
>>
>> Best
>>
>> Etienne
>>
>> On 11/03/2021 17:17, Etienne Chauchot wrote:
>>> Hi all,
>>>
>>> I just submitted another parquet PR that adds ParquetAvroInputFormat
>>> (I'm using it in a benchmark I'm coding). If anyone is interested in
>>> reviewing it, be my guest:
>>>
>>> https://github.com/apache/flink/pull/15156
>>>
>>> I have also an older parquet PR that fixes a format conversion bug
>>> that is waiting for merge if anyone can review it also (already 1
>>> approval of a non-committer, thanks @HuangZhenQiu
>>> <https://github.com/HuangZhenQiu>):
>>>
>>> https://github.com/apache/flink/pull/14961
>>>
>>> If I have time, I'll also tackle the other parquet tickets that I
>>> opened lately
>>>
>>> Best
>>>
>>> Etienne
>>>
>>> On 25/02/2021 08:34, Jingsong Li wrote:
>>>> Hi Etienne,
>>>>
>>>> ParquetColumnarRowInputFormat is not fully functional yet, it has a good
>>>> performance, but it is hard to support complex types, like array and
>> map...
>>>> So I think a migrated ParquetInputFormat version is required.
>>>>
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Wed, Feb 24, 2021 at 3:43 PM Etienne Chauchot<[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thanks guys for the comments !
>>>>>
>>>>> I did not know it was legacy. I will give the new sources a try.
>>>>>
>>>>> Jingsong, when you say "migrate ParquetInputFormat to the new
>> BulkFormat
>>>>> interface", do you mean that the new ParquetColumnarRowInputFormat is
>>>>> not fully functional yet?
>>>>>
>>>>> In the meantime, if you agree, I think I'm still gonna submit a PR for
>>>>> https://issues.apache.org/jira/browse/FLINK-21393 because I need it
>> on
>>>>> an urgent task I'm doing.
>>>>>
>>>>> Best
>>>>>
>>>>> Etienne
>>>>>
>>>>> On 24/02/2021 03:41, Peter Huang wrote:
>>>>>> Hi Jingsong,
>>>>>>
>>>>>> Thanks for pointing this out. Actually, I planned to work on changing
>>>>>> interfaces ParquetTableSource and ParquetInputFormat.
>>>>>> After refactoring the code, I may also help to fix the issue in
>>>>>> https://issues.apache.org/jira/browse/FLINK-21468.
>>>>>>
>>>>>> Best Regards
>>>>>> Peter Huang
>>>>>>
>>>>>> On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li<[hidden email]>
>>>>> wrote:
>>>>>>> Hi Etienne,
>>>>>>>
>>>>>>> Thanks for your reporting.
>>>>>>>
>>>>>>> There are indeed many problems. There is no doubt that we need to
>>>>> improve
>>>>>>> our current format implementation.
>>>>>>>
>>>>>>> But ParquetTableSource and ParquetInputFormat are legacy
>> implementations
>>>>>>> with legacy interfaces. We have introduced new interfaces for
>> execution
>>>>> and
>>>>>>> SQL. You can see:
>>>>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
>>>>> for
>>>>>>> columnar row reading, not support complex types, we need
>>>>>>> migrate ParquetInputFormat to the new BulkFormat interface.
>>>>>>> - FileSystemTableSource with DynamicTableSource interface, It is a
>>>>> generic
>>>>>>> FileSystem source for all formats, we can just use it for parquet
>> too.
>>>>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
>>>>>>> interfaces, I think we can finish migration work first, what do you
>>>>> think?
>>>>>>> Best,
>>>>>>> Jingsong
>>>>>>>
>>>>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <
>> [hidden email]
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I've been playing with Parquet with SQL and Avro lately. I've found
>>>>> some
>>>>>>>> bugs:
>>>>>>>>
>>>>>>>> 1.https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>>>>>>> submitted a PR on this one (
>> https://github.com/apache/flink/pull/14961
>>>>> )
>>>>>>>> 2.https://issues.apache.org/jira/browse/FLINK-21389
>>>>>>>>
>>>>>>>> 3.https://issues.apache.org/jira/browse/FLINK-21468
>>>>>>>>
>>>>>>>> I've already started to work on this ticket:
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>>>>>>
>>>>>>>>
>>>>>>>> I'd be happy to receive your comments on these tickets
>>>>>>>>
>>>>>>>>
>>>>>>>> Best
>>>>>>>>
>>>>>>>> Etienne Chauchot
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Best, Jingsong Lee
>>>>>>>
>

Etienne Chauchot

Re: [Parquet support]

Hi Jingsong,

I just submitted a cherry-pick PR
https://github.com/apache/flink/pull/15172 of (1) to release-.1.12 branch

[1] https://github.com/apache/flink/pull/14961

Etienne

On 12/03/2021 14:55, Etienne Chauchot wrote:

> Hi Jingsong,
>
> No problem for the delay. Thanks for merging the first parquet PR.
>
> I'll submit the 2 PRs to 1.12 when they're all merged to master. For
> that, I just have to submit a PR against this branch:
> https://github.com/apache/flink/tree/release-1.12 ?
>
> Best,
>
> Etienne
>
> On 12/03/2021 03:56, Jingsong Li wrote:
>> Hi Etienne,
>>
>> Sorry for the late reply,
>>
>> I just merged your bug fixing.
>> I think you can submit a PR for release-1.12.
>>
>> Best,
>> Jingsong
>>
>> On Fri, Mar 12, 2021 at 12:22 AM Etienne Chauchot <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I forgot to mention that I submitted the new ParquetAvroInputFormat to
>>> master (1.13) but it is made to work for 1.12.x (last release) also and
>>> I'm using it with Flink 1.12.x.
>>>
>>> Maybe it could be a good candidate to be included in an upcoming 1.12.3
>>> release, WDYT ?
>>>
>>> Best
>>>
>>> Etienne
>>>
>>> On 11/03/2021 17:17, Etienne Chauchot wrote:
>>>> Hi all,
>>>>
>>>> I just submitted another parquet PR that adds ParquetAvroInputFormat
>>>> (I'm using it in a benchmark I'm coding). If anyone is interested in
>>>> reviewing it, be my guest:
>>>>
>>>> https://github.com/apache/flink/pull/15156
>>>>
>>>> I have also an older parquet PR that fixes a format conversion bug
>>>> that is waiting for merge if anyone can review it also (already 1
>>>> approval of a non-committer, thanks @HuangZhenQiu
>>>> <https://github.com/HuangZhenQiu>):
>>>>
>>>> https://github.com/apache/flink/pull/14961
>>>>
>>>> If I have time, I'll also tackle the other parquet tickets that I
>>>> opened lately
>>>>
>>>> Best
>>>>
>>>> Etienne
>>>>
>>>> On 25/02/2021 08:34, Jingsong Li wrote:
>>>>> Hi Etienne,
>>>>>
>>>>> ParquetColumnarRowInputFormat is not fully functional yet, it has
>>>>> a good
>>>>> performance, but it is hard to support complex types, like array and
>>> map...
>>>>> So I think a migrated ParquetInputFormat version is required.
>>>>>
>>>>> Best,
>>>>> Jingsong
>>>>>
>>>>> On Wed, Feb 24, 2021 at 3:43 PM Etienne
>>>>> Chauchot<[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks guys for the comments !
>>>>>>
>>>>>> I did not know it was legacy. I will give the new sources a try.
>>>>>>
>>>>>> Jingsong, when you say "migrate ParquetInputFormat to the new
>>> BulkFormat
>>>>>> interface", do you mean that the new
>>>>>> ParquetColumnarRowInputFormat is
>>>>>> not fully functional yet?
>>>>>>
>>>>>> In the meantime, if you agree, I think I'm still gonna submit a
>>>>>> PR for
>>>>>> https://issues.apache.org/jira/browse/FLINK-21393 because I need it
>>> on
>>>>>> an urgent task I'm doing.
>>>>>>
>>>>>> Best
>>>>>>
>>>>>> Etienne
>>>>>>
>>>>>> On 24/02/2021 03:41, Peter Huang wrote:
>>>>>>> Hi Jingsong,
>>>>>>>
>>>>>>> Thanks for pointing this out. Actually, I planned to work on
>>>>>>> changing
>>>>>>> interfaces ParquetTableSource and ParquetInputFormat.
>>>>>>> After refactoring the code, I may also help to fix the issue in
>>>>>>> https://issues.apache.org/jira/browse/FLINK-21468.
>>>>>>>
>>>>>>> Best Regards
>>>>>>> Peter Huang
>>>>>>>
>>>>>>> On Tue, Feb 23, 2021 at 6:35 PM Jingsong Li<[hidden email]>
>>>>>> wrote:
>>>>>>>> Hi Etienne,
>>>>>>>>
>>>>>>>> Thanks for your reporting.
>>>>>>>>
>>>>>>>> There are indeed many problems. There is no doubt that we need to
>>>>>> improve
>>>>>>>> our current format implementation.
>>>>>>>>
>>>>>>>> But ParquetTableSource and ParquetInputFormat are legacy
>>> implementations
>>>>>>>> with legacy interfaces. We have introduced new interfaces for
>>> execution
>>>>>> and
>>>>>>>> SQL. You can see:
>>>>>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It
>>>>>>>> is just
>>>>>> for
>>>>>>>> columnar row reading, not support complex types, we need
>>>>>>>> migrate ParquetInputFormat to the new BulkFormat interface.
>>>>>>>> - FileSystemTableSource with DynamicTableSource interface, It is a
>>>>>> generic
>>>>>>>> FileSystem source for all formats, we can just use it for parquet
>>> too.
>>>>>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
>>>>>>>> interfaces, I think we can finish migration work first, what do
>>>>>>>> you
>>>>>> think?
>>>>>>>> Best,
>>>>>>>> Jingsong
>>>>>>>>
>>>>>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <
>>> [hidden email]
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I've been playing with Parquet with SQL and Avro lately. I've
>>>>>>>>> found
>>>>>> some
>>>>>>>>> bugs:
>>>>>>>>>
>>>>>>>>> 1.https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>>>>>>>> submitted a PR on this one (
>>> https://github.com/apache/flink/pull/14961
>>>>>> )
>>>>>>>>> 2.https://issues.apache.org/jira/browse/FLINK-21389
>>>>>>>>>
>>>>>>>>> 3.https://issues.apache.org/jira/browse/FLINK-21468
>>>>>>>>>
>>>>>>>>> I've already started to work on this ticket:
>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'd be happy to receive your comments on these tickets
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>>
>>>>>>>>> Etienne Chauchot
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Best, Jingsong Lee
>>>>>>>>
>>

Etienne Chauchot

Re: [Parquet support]

Hi all,

The fix (https://issues.apache.org/jira/browse/FLINK-21388) is now also
available for flink 1.12 also (thanks Jingsong for merging the
cherrypick PR)

But before releasing 1.12 branch, I'd like this other PR to be merged:
https://github.com/apache/flink/pull/15156 that introduces
ParquetAvroInputFormat.

In this PR I just added deep field copy (in case the source schema is
multi-level) and fixed serialization issues that I found testing on
flink 1.11. It should be ready for review.

Once it is merged I'll cherry pick to 1.12 branch.

=> When is the next 1.12 release scheduled ? Do we have enough time to
include this second parquet feature ?

Best

Etienne Chauchot

On 12/03/2021 15:31, Etienne Chauchot wrote:

> Hi Jingsong,
>
> I just submitted a cherry-pick PR
> https://github.com/apache/flink/pull/15172 of (1) to release-.1.12 branch
>
> [1] https://github.com/apache/flink/pull/14961
>
> Etienne
>
> On 12/03/2021 14:55, Etienne Chauchot wrote:
>> Hi Jingsong,
>>
>> No problem for the delay. Thanks for merging the first parquet PR.
>>
>> I'll submit the 2 PRs to 1.12 when they're all merged to master. For
>> that, I just have to submit a PR against this branch:
>> https://github.com/apache/flink/tree/release-1.12 ?
>>
>> Best,
>>
>> Etienne
>>
>> On 12/03/2021 03:56, Jingsong Li wrote:
>>> Hi Etienne,
>>>
>>> Sorry for the late reply,
>>>
>>> I just merged your bug fixing.
>>> I think you can submit a PR for release-1.12.
>>>
>>> Best,
>>> Jingsong
>>>
>>> On Fri, Mar 12, 2021 at 12:22 AM Etienne Chauchot
>>> <[hidden email]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I forgot to mention that I submitted the new ParquetAvroInputFormat to
>>>> master (1.13) but it is made to work for 1.12.x (last release) also
>>>> and
>>>> I'm using it with Flink 1.12.x.
>>>>
>>>> Maybe it could be a good candidate to be included in an upcoming
>>>> 1.12.3
>>>> release, WDYT ?
>>>>
>>>> Best
>>>>
>>>> Etienne
>>>>
>>>> On 11/03/2021 17:17, Etienne Chauchot wrote:
>>>>> Hi all,
>>>>>
>>>>> I just submitted another parquet PR that adds ParquetAvroInputFormat
>>>>> (I'm using it in a benchmark I'm coding). If anyone is interested in
>>>>> reviewing it, be my guest:
>>>>>
>>>>> https://github.com/apache/flink/pull/15156
>>>>>
>>>>> I have also an older parquet PR that fixes a format conversion bug
>>>>> that is waiting for merge if anyone can review it also (already 1
>>>>> approval of a non-committer, thanks @HuangZhenQiu
>>>>> <https://github.com/HuangZhenQiu>):
>>>>>
>>>>> https://github.com/apache/flink/pull/14961
>>>>>
>>>>> If I have time, I'll also tackle the other parquet tickets that I
>>>>> opened lately
>>>>>
>>>>> Best
>>>>>
>>>>> Etienne
>>>>>
>>>>> On 25/02/2021 08:34, Jingsong Li wrote:
>>>>>> Hi Etienne,
>>>>>>
>>>>>> ParquetColumnarRowInputFormat is not fully functional yet, it has
>>>>>> a good
>>>>>> performance, but it is hard to support complex types, like array and
>>>> map...
>>>>>> So I think a migrated ParquetInputFormat version is required.
>>>>>>
>>>>>> Best,
>>>>>> Jingsong
>>>>>>
>>>>>> On Wed, Feb 24, 2021 at 3:43 PM Etienne
>>>>>> Chauchot<[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Thanks guys for the comments !
>>>>>>>
>>>>>>> I did not know it was legacy. I will give the new sources a try.
>>>>>>>
>>>>>>> Jingsong, when you say "migrate ParquetInputFormat to the new
>>>> BulkFormat
>>>>>>> interface", do you mean that the new
>>>>>>> ParquetColumnarRowInputFormat is
>>>>>>> not fully functional yet?
>>>>>>>
>>>>>>> In the meantime, if you agree, I think I'm still gonna submit a
>>>>>>> PR for
>>>>>>> https://issues.apache.org/jira/browse/FLINK-21393 because I need it
>>>> on
>>>>>>> an urgent task I'm doing.
>>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>> Etienne
>>>>>>>
>>>>>>> On 24/02/2021 03:41, Peter Huang wrote:
>>>>>>>> Hi Jingsong,
>>>>>>>>
>>>>>>>> Thanks for pointing this out. Actually, I planned to work on
>>>>>>>> changing
>>>>>>>> interfaces ParquetTableSource and ParquetInputFormat.
>>>>>>>> After refactoring the code, I may also help to fix the issue in
>>>>>>>> https://issues.apache.org/jira/browse/FLINK-21468.
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>> Peter Huang
>>>>>>>>
>>>>>>>> On Tue, Feb 23, 2021 at 6:35 PM Jingsong
>>>>>>>> Li<[hidden email]>
>>>>>>> wrote:
>>>>>>>>> Hi Etienne,
>>>>>>>>>
>>>>>>>>> Thanks for your reporting.
>>>>>>>>>
>>>>>>>>> There are indeed many problems. There is no doubt that we need to
>>>>>>> improve
>>>>>>>>> our current format implementation.
>>>>>>>>>
>>>>>>>>> But ParquetTableSource and ParquetInputFormat are legacy
>>>> implementations
>>>>>>>>> with legacy interfaces. We have introduced new interfaces for
>>>> execution
>>>>>>> and
>>>>>>>>> SQL. You can see:
>>>>>>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It
>>>>>>>>> is just
>>>>>>> for
>>>>>>>>> columnar row reading, not support complex types, we need
>>>>>>>>> migrate ParquetInputFormat to the new BulkFormat interface.
>>>>>>>>> - FileSystemTableSource with DynamicTableSource interface, It
>>>>>>>>> is a
>>>>>>> generic
>>>>>>>>> FileSystem source for all formats, we can just use it for parquet
>>>> too.
>>>>>>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
>>>>>>>>> interfaces, I think we can finish migration work first, what
>>>>>>>>> do you
>>>>>>> think?
>>>>>>>>> Best,
>>>>>>>>> Jingsong
>>>>>>>>>
>>>>>>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <
>>>> [hidden email]
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I've been playing with Parquet with SQL and Avro lately. I've
>>>>>>>>>> found
>>>>>>> some
>>>>>>>>>> bugs:
>>>>>>>>>>
>>>>>>>>>> 1.https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>>>>>>>>> submitted a PR on this one (
>>>> https://github.com/apache/flink/pull/14961
>>>>>>> )
>>>>>>>>>> 2.https://issues.apache.org/jira/browse/FLINK-21389
>>>>>>>>>>
>>>>>>>>>> 3.https://issues.apache.org/jira/browse/FLINK-21468
>>>>>>>>>>
>>>>>>>>>> I've already started to work on this ticket:
>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'd be happy to receive your comments on these tickets
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best
>>>>>>>>>>
>>>>>>>>>> Etienne Chauchot
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best, Jingsong Lee
>>>>>>>>>
>>>

Etienne Chauchot

Re: [Parquet support]

In reply to this post by Jingsong Li

Hi,

@Jingsong, I agree that adding a new feature (ParquetAvroInputFormat) to
an old source API is a maintenance burden. But IMHO I think that while
the new DataStream batch/streaming convergent API is not 100% functional
we still need to maintain older sources and add missing features to them.

Indeed, I realized that DataStream API in batch mode (1) does not
support aggregations yet (2) so in such a case a user would stick to the
DataSet API. And the new FileSource API with
ParquetColumnarRowInputFormat is not available in DataSet API (3).

So, long story short, in some cases a user will have no other choice
than using ParquetInputFormat and legacy source.

WDYT ?

[1] https://issues.apache.org/jira/browse/FLINK-19316

[2] https://issues.apache.org/jira/browse/FLINK-22587

[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-Compatibility,Deprecation,andMigrationPlan

Best,

Etienne

On 24/02/2021 03:35, Jingsong Li wrote:

> Hi Etienne,
>
> Thanks for your reporting.
>
> There are indeed many problems. There is no doubt that we need to improve
> our current format implementation.
>
> But ParquetTableSource and ParquetInputFormat are legacy implementations
> with legacy interfaces. We have introduced new interfaces for execution and
> SQL. You can see:
> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just for
> columnar row reading, not support complex types, we need
> migrate ParquetInputFormat to the new BulkFormat interface.
> - FileSystemTableSource with DynamicTableSource interface, It is a generic
> FileSystem source for all formats, we can just use it for parquet too.
>
> Considering ParquetTableSource and ParquetInputFormat are legacy
> interfaces, I think we can finish migration work first, what do you think?
>
> Best,
> Jingsong
>
> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]>
> wrote:
>
>> Hi all,
>>
>> I've been playing with Parquet with SQL and Avro lately. I've found some
>> bugs:
>>
>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
>> submitted a PR on this one (https://github.com/apache/flink/pull/14961)
>>
>> 2. https://issues.apache.org/jira/browse/FLINK-21389
>>
>> 3. https://issues.apache.org/jira/browse/FLINK-21468
>>
>> I've already started to work on this ticket:
>> https://issues.apache.org/jira/browse/FLINK-21393
>>
>>
>> I'd be happy to receive your comments on these tickets
>>
>>
>> Best
>>
>> Etienne Chauchot
>>
>>
>>

Etienne Chauchot

Re: [Parquet support]

Hi all,

considering (see my email below) that the DataStream API is not fully
functional yet (batch mode) and considering that the new sources are
only available on DataStream API, can we merge this PR (1) about
existing sources in DataSet API? It has received already one LGTM.

[1] https://github.com/apache/flink/pull/15156

anyone ?

Etienne

On 06/05/2021 14:23, Etienne Chauchot wrote:

> Hi,
>
> @Jingsong, I agree that adding a new feature (ParquetAvroInputFormat)
> to an old source API is a maintenance burden. But IMHO I think that
> while the new DataStream batch/streaming convergent API is not 100%
> functional we still need to maintain older sources and add missing
> features to them.
>
> Indeed, I realized that DataStream API in batch mode (1) does not
> support aggregations yet (2) so in such a case a user would stick to
> the DataSet API. And the new FileSource API with
> ParquetColumnarRowInputFormat is not available in DataSet API (3).
>
> So, long story short, in some cases a user will have no other choice
> than using ParquetInputFormat and legacy source.
>
> WDYT ?
>
> [1] https://issues.apache.org/jira/browse/FLINK-19316
>
> [2] https://issues.apache.org/jira/browse/FLINK-22587
>
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-Compatibility,Deprecation,andMigrationPlan
>
> Best,
>
> Etienne
>
> On 24/02/2021 03:35, Jingsong Li wrote:
>> Hi Etienne,
>>
>> Thanks for your reporting.
>>
>> There are indeed many problems. There is no doubt that we need to
>> improve
>> our current format implementation.
>>
>> But ParquetTableSource and ParquetInputFormat are legacy implementations
>> with legacy interfaces. We have introduced new interfaces for
>> execution and
>> SQL. You can see:
>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
>> for
>> columnar row reading, not support complex types, we need
>> migrate ParquetInputFormat to the new BulkFormat interface.
>> - FileSystemTableSource with DynamicTableSource interface, It is a
>> generic
>> FileSystem source for all formats, we can just use it for parquet too.
>>
>> Considering ParquetTableSource and ParquetInputFormat are legacy
>> interfaces, I think we can finish migration work first, what do you
>> think?
>>
>> Best,
>> Jingsong
>>
>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]>
>> wrote:
>>
>>> Hi all,
>>>
>>> I've been playing with Parquet with SQL and Avro lately. I've found
>>> some
>>> bugs:
>>>
>>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>> submitted a PR on this one (https://github.com/apache/flink/pull/14961)
>>>
>>> 2. https://issues.apache.org/jira/browse/FLINK-21389
>>>
>>> 3. https://issues.apache.org/jira/browse/FLINK-21468
>>>
>>> I've already started to work on this ticket:
>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>
>>>
>>> I'd be happy to receive your comments on these tickets
>>>
>>>
>>> Best
>>>
>>> Etienne Chauchot
>>>
>>>
>>>

Arvid Heise-4

Re: [Parquet support]

Hi Etienne,

I'm taking over and just left you a review. Sorry for the long delays.

Best,

Arvid

On Fri, May 21, 2021 at 11:25 AM Etienne Chauchot <[hidden email]>
wrote:

> Hi all,
>
> considering (see my email below) that the DataStream API is not fully
> functional yet (batch mode) and considering that the new sources are
> only available on DataStream API, can we merge this PR (1) about
> existing sources in DataSet API? It has received already one LGTM.
>
> [1] https://github.com/apache/flink/pull/15156
>
> anyone ?
>
> Etienne
>
> On 06/05/2021 14:23, Etienne Chauchot wrote:
> > Hi,
> >
> > @Jingsong, I agree that adding a new feature (ParquetAvroInputFormat)
> > to an old source API is a maintenance burden. But IMHO I think that
> > while the new DataStream batch/streaming convergent API is not 100%
> > functional we still need to maintain older sources and add missing
> > features to them.
> >
> > Indeed, I realized that DataStream API in batch mode (1) does not
> > support aggregations yet (2) so in such a case a user would stick to
> > the DataSet API. And the new FileSource API with
> > ParquetColumnarRowInputFormat is not available in DataSet API (3).
> >
> > So, long story short, in some cases a user will have no other choice
> > than using ParquetInputFormat and legacy source.
> >
> > WDYT ?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-19316
> >
> > [2] https://issues.apache.org/jira/browse/FLINK-22587
> >
> > [3]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-Compatibility,Deprecation,andMigrationPlan
> >
> > Best,
> >
> > Etienne
> >
> > On 24/02/2021 03:35, Jingsong Li wrote:
> >> Hi Etienne,
> >>
> >> Thanks for your reporting.
> >>
> >> There are indeed many problems. There is no doubt that we need to
> >> improve
> >> our current format implementation.
> >>
> >> But ParquetTableSource and ParquetInputFormat are legacy implementations
> >> with legacy interfaces. We have introduced new interfaces for
> >> execution and
> >> SQL. You can see:
> >> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
> >> for
> >> columnar row reading, not support complex types, we need
> >> migrate ParquetInputFormat to the new BulkFormat interface.
> >> - FileSystemTableSource with DynamicTableSource interface, It is a
> >> generic
> >> FileSystem source for all formats, we can just use it for parquet too.
> >>
> >> Considering ParquetTableSource and ParquetInputFormat are legacy
> >> interfaces, I think we can finish migration work first, what do you
> >> think?
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]
> >
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I've been playing with Parquet with SQL and Avro lately. I've found
> >>> some
> >>> bugs:
> >>>
> >>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
> >>> submitted a PR on this one (https://github.com/apache/flink/pull/14961
> )
> >>>
> >>> 2. https://issues.apache.org/jira/browse/FLINK-21389
> >>>
> >>> 3. https://issues.apache.org/jira/browse/FLINK-21468
> >>>
> >>> I've already started to work on this ticket:
> >>> https://issues.apache.org/jira/browse/FLINK-21393
> >>>
> >>>
> >>> I'd be happy to receive your comments on these tickets
> >>>
> >>>
> >>> Best
> >>>
> >>> Etienne Chauchot
> >>>
> >>>
> >>>
>

Etienne Chauchot

Re: [Parquet support]

Thanks Arvid !

Etienne

On 21/05/2021 12:45, Arvid Heise wrote:

> Hi Etienne,
>
> I'm taking over and just left you a review. Sorry for the long delays.
>
> Best,
>
> Arvid
>
> On Fri, May 21, 2021 at 11:25 AM Etienne Chauchot <[hidden email]>
> wrote:
>
>> Hi all,
>>
>> considering (see my email below) that the DataStream API is not fully
>> functional yet (batch mode) and considering that the new sources are
>> only available on DataStream API, can we merge this PR (1) about
>> existing sources in DataSet API? It has received already one LGTM.
>>
>> [1] https://github.com/apache/flink/pull/15156
>>
>> anyone ?
>>
>> Etienne
>>
>> On 06/05/2021 14:23, Etienne Chauchot wrote:
>>> Hi,
>>>
>>> @Jingsong, I agree that adding a new feature (ParquetAvroInputFormat)
>>> to an old source API is a maintenance burden. But IMHO I think that
>>> while the new DataStream batch/streaming convergent API is not 100%
>>> functional we still need to maintain older sources and add missing
>>> features to them.
>>>
>>> Indeed, I realized that DataStream API in batch mode (1) does not
>>> support aggregations yet (2) so in such a case a user would stick to
>>> the DataSet API. And the new FileSource API with
>>> ParquetColumnarRowInputFormat is not available in DataSet API (3).
>>>
>>> So, long story short, in some cases a user will have no other choice
>>> than using ParquetInputFormat and legacy source.
>>>
>>> WDYT ?
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-19316
>>>
>>> [2] https://issues.apache.org/jira/browse/FLINK-22587
>>>
>>> [3]
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface#FLIP27:RefactorSourceInterface-Compatibility,Deprecation,andMigrationPlan
>>> Best,
>>>
>>> Etienne
>>>
>>> On 24/02/2021 03:35, Jingsong Li wrote:
>>>> Hi Etienne,
>>>>
>>>> Thanks for your reporting.
>>>>
>>>> There are indeed many problems. There is no doubt that we need to
>>>> improve
>>>> our current format implementation.
>>>>
>>>> But ParquetTableSource and ParquetInputFormat are legacy implementations
>>>> with legacy interfaces. We have introduced new interfaces for
>>>> execution and
>>>> SQL. You can see:
>>>> - ParquetColumnarRowInputFormat with BulkFormat interface. It is just
>>>> for
>>>> columnar row reading, not support complex types, we need
>>>> migrate ParquetInputFormat to the new BulkFormat interface.
>>>> - FileSystemTableSource with DynamicTableSource interface, It is a
>>>> generic
>>>> FileSystem source for all formats, we can just use it for parquet too.
>>>>
>>>> Considering ParquetTableSource and ParquetInputFormat are legacy
>>>> interfaces, I think we can finish migration work first, what do you
>>>> think?
>>>>
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Wed, Feb 24, 2021 at 12:46 AM Etienne Chauchot <[hidden email]
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I've been playing with Parquet with SQL and Avro lately. I've found
>>>>> some
>>>>> bugs:
>>>>>
>>>>> 1. https://issues.apache.org/jira/browse/FLINK-21388 : I already
>>>>> submitted a PR on this one (https://github.com/apache/flink/pull/14961
>> )
>>>>> 2. https://issues.apache.org/jira/browse/FLINK-21389
>>>>>
>>>>> 3. https://issues.apache.org/jira/browse/FLINK-21468
>>>>>
>>>>> I've already started to work on this ticket:
>>>>> https://issues.apache.org/jira/browse/FLINK-21393
>>>>>
>>>>>
>>>>> I'd be happy to receive your comments on these tickets
>>>>>
>>>>>
>>>>> Best
>>>>>
>>>>> Etienne Chauchot
>>>>>
>>>>>
>>>>>