Creating TemporalTable based on Catalog table in SQL Client

[Gyula Fóra]

Hi all!

I was testing the TemporalTable functionality in the SQL client while using
the Hive Catalog and I ran into the following problem.

I have a table created in the Hive catalog and I want to create a temporal
table over it.

As we cannot create temporal tables in SQL directly I have to define it in
the environment yaml file. Unfortunately it seems to be impossible to
reference a table only present in the catalog (not in the yaml) as catalogs
are loaded only after creating the temporal table (see
https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
)

I am wondering if it would make sense to set the catalogs before all else
or if that would cause some other problems.

What do you think?
Gyula

[bowen.li]

Hi Gyula,

What line 622 (the link you shared) does is not registering catalogs, but
setting an already registered catalog as the current one. As you can see
from the method and its comment, catalogs are loaded first before any
tables in yaml are registered, so you should be able to achieve what you
described.

Bowen

On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra <[hidden email]> wrote:

> Hi all!
>
> I was testing the TemporalTable functionality in the SQL client while using
> the Hive Catalog and I ran into the following problem.
>
> I have a table created in the Hive catalog and I want to create a temporal
> table over it.
>
> As we cannot create temporal tables in SQL directly I have to define it in
> the environment yaml file. Unfortunately it seems to be impossible to
> reference a table only present in the catalog (not in the yaml) as catalogs
> are loaded only after creating the temporal table (see
>
> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
> )
>
> I am wondering if it would make sense to set the catalogs before all else
> or if that would cause some other problems.
>
> What do you think?
> Gyula
>

[Gyula Fóra]

You are right but still if the default catalog is something else and that's
the one containing the table then it still wont work currently.

Gyula

On Wed, Mar 4, 2020 at 5:08 AM Bowen Li <[hidden email]> wrote:

> Hi Gyula,
>
> What line 622 (the link you shared) does is not registering catalogs, but
> setting an already registered catalog as the current one. As you can see
> from the method and its comment, catalogs are loaded first before any
> tables in yaml are registered, so you should be able to achieve what you
> described.
>
> Bowen
>
> On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra <[hidden email]> wrote:
>
> > Hi all!
> >
> > I was testing the TemporalTable functionality in the SQL client while
> using
> > the Hive Catalog and I ran into the following problem.
> >
> > I have a table created in the Hive catalog and I want to create a
> temporal
> > table over it.
> >
> > As we cannot create temporal tables in SQL directly I have to define it
> in
> > the environment yaml file. Unfortunately it seems to be impossible to
> > reference a table only present in the catalog (not in the yaml) as
> catalogs
> > are loaded only after creating the temporal table (see
> >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
> > )
> >
> > I am wondering if it would make sense to set the catalogs before all else
> > or if that would cause some other problems.
> >
> > What do you think?
> > Gyula
> >
>

[Gyula Fóra]

I guess it will only work now if you specify the catalog name too when
referencing the table.


On Wed, Mar 4, 2020 at 11:15 AM Gyula Fóra <[hidden email]> wrote:

> You are right but still if the default catalog is something else and
> that's the one containing the table then it still wont work currently.
>
> Gyula
>
> On Wed, Mar 4, 2020 at 5:08 AM Bowen Li <[hidden email]> wrote:
>
>> Hi Gyula,
>>
>> What line 622 (the link you shared) does is not registering catalogs, but
>> setting an already registered catalog as the current one. As you can see
>> from the method and its comment, catalogs are loaded first before any
>> tables in yaml are registered, so you should be able to achieve what you
>> described.
>>
>> Bowen
>>
>> On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra <[hidden email]> wrote:
>>
>> > Hi all!
>> >
>> > I was testing the TemporalTable functionality in the SQL client while
>> using
>> > the Hive Catalog and I ran into the following problem.
>> >
>> > I have a table created in the Hive catalog and I want to create a
>> temporal
>> > table over it.
>> >
>> > As we cannot create temporal tables in SQL directly I have to define it
>> in
>> > the environment yaml file. Unfortunately it seems to be impossible to
>> > reference a table only present in the catalog (not in the yaml) as
>> catalogs
>> > are loaded only after creating the temporal table (see
>> >
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
>> > )
>> >
>> > I am wondering if it would make sense to set the catalogs before all
>> else
>> > or if that would cause some other problems.
>> >
>> > What do you think?
>> > Gyula
>> >
>>
>

[bowen.li]

you would need to reference the table with fully qualified name with
catalog and database

On Wed, Mar 4, 2020 at 02:17 Gyula Fóra <[hidden email]> wrote:

> I guess it will only work now if you specify the catalog name too when
> referencing the table.
>
>
> On Wed, Mar 4, 2020 at 11:15 AM Gyula Fóra <[hidden email]> wrote:
>
> > You are right but still if the default catalog is something else and
> > that's the one containing the table then it still wont work currently.
> >
> > Gyula
> >
> > On Wed, Mar 4, 2020 at 5:08 AM Bowen Li <[hidden email]> wrote:
> >
> >> Hi Gyula,
> >>
> >> What line 622 (the link you shared) does is not registering catalogs,
> but
> >> setting an already registered catalog as the current one. As you can see
> >> from the method and its comment, catalogs are loaded first before any
> >> tables in yaml are registered, so you should be able to achieve what you
> >> described.
> >>
> >> Bowen
> >>
> >> On Tue, Mar 3, 2020 at 5:16 AM Gyula Fóra <[hidden email]> wrote:
> >>
> >> > Hi all!
> >> >
> >> > I was testing the TemporalTable functionality in the SQL client while
> >> using
> >> > the Hive Catalog and I ran into the following problem.
> >> >
> >> > I have a table created in the Hive catalog and I want to create a
> >> temporal
> >> > table over it.
> >> >
> >> > As we cannot create temporal tables in SQL directly I have to define
> it
> >> in
> >> > the environment yaml file. Unfortunately it seems to be impossible to
> >> > reference a table only present in the catalog (not in the yaml) as
> >> catalogs
> >> > are loaded only after creating the temporal table (see
> >> >
> >> >
> >>
> https://github.com/apache/flink/blob/master/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/gateway/local/ExecutionContext.java#L622
> >> > )
> >> >
> >> > I am wondering if it would make sense to set the catalogs before all
> >> else
> >> > or if that would cause some other problems.
> >> >
> >> > What do you think?
> >> > Gyula
> >> >
> >>
> >
>