Fabian Hueske created FLINK-7696:
------------------------------------
Summary: Add projection push-down support for TableSources with time attributes
Key: FLINK-7696
URL:
https://issues.apache.org/jira/browse/FLINK-7696 Project: Flink
Issue Type: Improvement
Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: Fabian Hueske
Table sources that implement the {{DefinedProctimeAttribute}} or {{DefinedRowtimeAttribute}} do not support projection push-down even if they also implement {{ProjectableTableSource}}.
There are several problems:
- the schema of a {{TableSource}} that implements {{DefinedRowtimeAttribute}} or {{DefinedProctimeAttribute}} is constructed in the catalog not in the {{TableSource}} (proctime fields are always appended at the end).
- the {{ProjectableTableSource.projectFields()}} method returns the projected fields as int indicies. In order to handle the indicies correctly, the TableSource would need to know about the internals of the Table API.
- {{ProjectableTableSource.projectFields()}} might reorder fields and move a proctime field into the middle of the schema. However, the TableSource has no control over that.
- A {{TableSource}} that implements {{DefinedRowtimeAttribute}} or {{DefinedProctimeAttribute}} would need to change the return values of {{getRowtimeAttribute()}} or {{getProctimeAttribute()}} depending on whether the attribute is kept or not.
Adjusting the schema of table sources inside of the Table API makes all of this quite messy. Maybe we need to refine the interfaces. For instance, we could ask users to explicitly add time indicator fields in the {{TypeInformation}} returned by {{TableSource.getReturnType()}}. However, that might collide with plans to add computable time attributes as proposed in FLINK-7548.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Locked
