[jira] [Created] (FLINK-15644) Add support for SQL query validation

[Shang Yuanchun (Jira)]

Fabian Hueske created FLINK-15644:
-------------------------------------

             Summary: Add support for SQL query validation
                 Key: FLINK-15644
                 URL: https://issues.apache.org/jira/browse/FLINK-15644
             Project: Flink
          Issue Type: New Feature
          Components: Table SQL / API
            Reporter: Fabian Hueske


It would be good if the {{TableEnvironment}} would offer methods to check the validity of SQL queries. Such a method could be used by services (CLI query shells, notebooks, SQL UIs) that are backed by Flink and execute their queries on Flink.

Validation should be available in two levels:
 # Validation of syntax and semantics: This includes parsing the query, checking the catalog for dbs, tables, fields, type checks for expressions and functions, etc. This will check if the query is a valid SQL query.
 # Validation that query is supported: Checks if Flink can execute the given query. Some syntactically and semantically valid SQL queries are not supported, esp. in a streaming context. This requires running the optimizer. If the optimizer generates an execution plan, the query can be executed. This check includes the first step and is more expensive.

The reason for this separation is that the first check can be done much fast as it does not involve calling the optimizer. Hence, it would be suitable for fast checks in an interactive query editor. The second check might take more time (depending on the complexity of the query) and might not be suitable for rapid checks but only on explicit user request.

Requirements:
 * validation does not modify the state of the {{TableEnvironment}}, i.e. it does not add plan operators
 * validation does not require connector dependencies
 * validation can identify the update mode of a continuous query result (append-only, upsert, retraction).

Out of scope for this issue:
 * better error messages for unsupported features as suggested by FLINK-7217



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[Flavio Pompermaier]

Why not adding also a suggest() method (also unimplemented initially) that
would return the list of suitable completions/tokens on the current query?
How complex eould it be to implement it in you opinion?

Il Ven 17 Gen 2020, 18:32 Fabian Hueske (Jira) <[hidden email]> ha scritto:

> Fabian Hueske created FLINK-15644:
> -------------------------------------
>
>              Summary: Add support for SQL query validation
>                  Key: FLINK-15644
>                  URL: https://issues.apache.org/jira/browse/FLINK-15644
>              Project: Flink
>           Issue Type: New Feature
>           Components: Table SQL / API
>             Reporter: Fabian Hueske
>
>
> It would be good if the {{TableEnvironment}} would offer methods to check
> the validity of SQL queries. Such a method could be used by services (CLI
> query shells, notebooks, SQL UIs) that are backed by Flink and execute
> their queries on Flink.
>
> Validation should be available in two levels:
>  # Validation of syntax and semantics: This includes parsing the query,
> checking the catalog for dbs, tables, fields, type checks for expressions
> and functions, etc. This will check if the query is a valid SQL query.
>  # Validation that query is supported: Checks if Flink can execute the
> given query. Some syntactically and semantically valid SQL queries are not
> supported, esp. in a streaming context. This requires running the
> optimizer. If the optimizer generates an execution plan, the query can be
> executed. This check includes the first step and is more expensive.
>
> The reason for this separation is that the first check can be done much
> fast as it does not involve calling the optimizer. Hence, it would be
> suitable for fast checks in an interactive query editor. The second check
> might take more time (depending on the complexity of the query) and might
> not be suitable for rapid checks but only on explicit user request.
>
> Requirements:
>  * validation does not modify the state of the {{TableEnvironment}}, i.e.
> it does not add plan operators
>  * validation does not require connector dependencies
>  * validation can identify the update mode of a continuous query result
> (append-only, upsert, retraction).
>
> Out of scope for this issue:
>  * better error messages for unsupported features as suggested by
> FLINK-7217
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

[godfreyhe]

hi Flavio, TableEnvironment.getCompletionHints maybe already meet the
requirement.

Flavio Pompermaier <[hidden email]> 于2020年1月18日周六 下午3:39写道：

> Why not adding also a suggest() method (also unimplemented initially) that
> would return the list of suitable completions/tokens on the current query?
> How complex eould it be to implement it in you opinion?
>
> Il Ven 17 Gen 2020, 18:32 Fabian Hueske (Jira) <[hidden email]> ha
> scritto:
>
> > Fabian Hueske created FLINK-15644:
> > -------------------------------------
> >
> >              Summary: Add support for SQL query validation
> >                  Key: FLINK-15644
> >                  URL: https://issues.apache.org/jira/browse/FLINK-15644
> >              Project: Flink
> >           Issue Type: New Feature
> >           Components: Table SQL / API
> >             Reporter: Fabian Hueske
> >
> >
> > It would be good if the {{TableEnvironment}} would offer methods to check
> > the validity of SQL queries. Such a method could be used by services (CLI
> > query shells, notebooks, SQL UIs) that are backed by Flink and execute
> > their queries on Flink.
> >
> > Validation should be available in two levels:
> >  # Validation of syntax and semantics: This includes parsing the query,
> > checking the catalog for dbs, tables, fields, type checks for expressions
> > and functions, etc. This will check if the query is a valid SQL query.
> >  # Validation that query is supported: Checks if Flink can execute the
> > given query. Some syntactically and semantically valid SQL queries are
> not
> > supported, esp. in a streaming context. This requires running the
> > optimizer. If the optimizer generates an execution plan, the query can be
> > executed. This check includes the first step and is more expensive.
> >
> > The reason for this separation is that the first check can be done much
> > fast as it does not involve calling the optimizer. Hence, it would be
> > suitable for fast checks in an interactive query editor. The second check
> > might take more time (depending on the complexity of the query) and might
> > not be suitable for rapid checks but only on explicit user request.
> >
> > Requirements:
> >  * validation does not modify the state of the {{TableEnvironment}}, i.e.
> > it does not add plan operators
> >  * validation does not require connector dependencies
> >  * validation can identify the update mode of a continuous query result
> > (append-only, upsert, retraction).
> >
> > Out of scope for this issue:
> >  * better error messages for unsupported features as suggested by
> > FLINK-7217
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
>

[Flavio Pompermaier]

Ok thanks for the pointer, I wasn't awareof that!

Il Dom 19 Gen 2020, 03:00 godfrey he <[hidden email]> ha scritto:

> hi Flavio, TableEnvironment.getCompletionHints maybe already meet the
> requirement.
>
> Flavio Pompermaier <[hidden email]> 于2020年1月18日周六 下午3:39写道：
>
> > Why not adding also a suggest() method (also unimplemented initially)
> that
> > would return the list of suitable completions/tokens on the current
> query?
> > How complex eould it be to implement it in you opinion?
> >
> > Il Ven 17 Gen 2020, 18:32 Fabian Hueske (Jira) <[hidden email]> ha
> > scritto:
> >
> > > Fabian Hueske created FLINK-15644:
> > > -------------------------------------
> > >
> > >              Summary: Add support for SQL query validation
> > >                  Key: FLINK-15644
> > >                  URL:
> https://issues.apache.org/jira/browse/FLINK-15644
> > >              Project: Flink
> > >           Issue Type: New Feature
> > >           Components: Table SQL / API
> > >             Reporter: Fabian Hueske
> > >
> > >
> > > It would be good if the {{TableEnvironment}} would offer methods to
> check
> > > the validity of SQL queries. Such a method could be used by services
> (CLI
> > > query shells, notebooks, SQL UIs) that are backed by Flink and execute
> > > their queries on Flink.
> > >
> > > Validation should be available in two levels:
> > >  # Validation of syntax and semantics: This includes parsing the query,
> > > checking the catalog for dbs, tables, fields, type checks for
> expressions
> > > and functions, etc. This will check if the query is a valid SQL query.
> > >  # Validation that query is supported: Checks if Flink can execute the
> > > given query. Some syntactically and semantically valid SQL queries are
> > not
> > > supported, esp. in a streaming context. This requires running the
> > > optimizer. If the optimizer generates an execution plan, the query can
> be
> > > executed. This check includes the first step and is more expensive.
> > >
> > > The reason for this separation is that the first check can be done much
> > > fast as it does not involve calling the optimizer. Hence, it would be
> > > suitable for fast checks in an interactive query editor. The second
> check
> > > might take more time (depending on the complexity of the query) and
> might
> > > not be suitable for rapid checks but only on explicit user request.
> > >
> > > Requirements:
> > >  * validation does not modify the state of the {{TableEnvironment}},
> i.e.
> > > it does not add plan operators
> > >  * validation does not require connector dependencies
> > >  * validation can identify the update mode of a continuous query result
> > > (append-only, upsert, retraction).
> > >
> > > Out of scope for this issue:
> > >  * better error messages for unsupported features as suggested by
> > > FLINK-7217
> > >
> > >
> > >
> > > --
> > > This message was sent by Atlassian Jira
> > > (v8.3.4#803005)
> > >
> >
>