[FLINK-3848] Add ProjectableTableSource

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[FLINK-3848] Add ProjectableTableSource

Anton Solovev
Hi folks,

Can you explain me what ProjectableTableSource is ?
What is it being invented for?

Best,
Anton

Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-3848] Add ProjectableTableSource

Fabian Hueske-2
Hi Anton,

a regular TableSource does not accept a predicate and return the whole
table.
A ProjectableTableSource is able to evaluate a predicate while scanning.
TableSources that evaluate predicates while (or rather before) scanning can
significantly reduce IO compared to a full scan.

Sources that could be access with a ProjectableTableSource would be a JDBC
database which pushes a query to a JDBC connected database, a Parquet file,
or an ORC file.

Best, Fabian






2016-11-02 13:53 GMT+01:00 Anton Solovev <[hidden email]>:

> Hi folks,
>
> Can you explain me what ProjectableTableSource is ?
> What is it being invented for?
>
> Best,
> Anton
>
>
Reply | Threaded
Open this post in threaded view
|

RE: [FLINK-3848] Add ProjectableTableSource

Anton Solovev
I have a question about type info, it looks like one in calcite mail you showed in jira,
but I catch CodeGenException when checking predefined table field types with output types from source

Flink cannot generate conversion because of row arity. We cannot reduce number of table fields,
but we can create a new code generation
or let nulls come from CsvInputFormt

Any ideas?

-----Original Message-----
From: Fabian Hueske [mailto:[hidden email]]
Sent: Wednesday, November 2, 2016 5:11 PM
To: [hidden email]
Subject: Re: [FLINK-3848] Add ProjectableTableSource

Hi Anton,

a regular TableSource does not accept a predicate and return the whole table.
A ProjectableTableSource is able to evaluate a predicate while scanning.
TableSources that evaluate predicates while (or rather before) scanning can significantly reduce IO compared to a full scan.

Sources that could be access with a ProjectableTableSource would be a JDBC database which pushes a query to a JDBC connected database, a Parquet file, or an ORC file.

Best, Fabian






2016-11-02 13:53 GMT+01:00 Anton Solovev <[hidden email]>:

> Hi folks,
>
> Can you explain me what ProjectableTableSource is ?
> What is it being invented for?
>
> Best,
> Anton
>
>