Support for out-of-the-box external catalog for SQL Client

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Support for out-of-the-box external catalog for SQL Client

Rong Rong
Hi everyone,

I was wondering if it is a good idea to support some external catalog
software, such as Apache HCatalog[2], out-of-the-box for the FLIP-24
SQL-Client[1]. There are many widely used catalogs that we can incorporate.
This way users won't have to always extend and create their own
ExternalCatalog.class separately and this could potentially make the
configuration part easier for SQL users.

Thanks,
Rong


[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
[2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
Reply | Threaded
Open this post in threaded view
|

Re: Support for out-of-the-box external catalog for SQL Client

Peter Huang
Hi Rong,

It is a good point out. I aligned with Fabian yesterday. It is a good work
that I can involve
to contribute back to Apache Flink after having AthenaX backfill support
internally.


Best Regards
Peter Huang

On Wed, Apr 11, 2018 at 10:52 AM, Rong Rong <[hidden email]> wrote:

> Hi everyone,
>
> I was wondering if it is a good idea to support some external catalog
> software, such as Apache HCatalog[2], out-of-the-box for the FLIP-24
> SQL-Client[1]. There are many widely used catalogs that we can incorporate.
> This way users won't have to always extend and create their own
> ExternalCatalog.class separately and this could potentially make the
> configuration part easier for SQL users.
>
> Thanks,
> Rong
>
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
> [2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
>
Reply | Threaded
Open this post in threaded view
|

Re: Support for out-of-the-box external catalog for SQL Client

Shuyi Chen
Thanks a lot, Rong and Peter.

AFAIK, there is a flink hcatalog connector introduced in FLINK-1466
<https://issues.apache.org/jira/browse/FLINK-1466> that is added by Fabian.
And there is another JIRA in FLINK-1913
<https://issues.apache.org/jira/browse/FLINK-1913> to document the use of
connector.

I think we can start with looking at the existing hcatalog  connector,
adding missing documentation, and come up with a proposal to evolve the
Flink HCatalog integration with ExternalCatalog, and the SQL client to make
it both useful both SQL and non-SQL scenarios.

Given we already have the integration implemented in AthenaX
<https://github.com/uber/AthenaX> internally, we can help drive and
contribute back to the community.

Shuyi

On Wed, Apr 11, 2018 at 12:01 PM, Peter Huang <[hidden email]>
wrote:

> Hi Rong,
>
> It is a good point out. I aligned with Fabian yesterday. It is a good work
> that I can involve
> to contribute back to Apache Flink after having AthenaX backfill support
> internally.
>
>
> Best Regards
> Peter Huang
>
> On Wed, Apr 11, 2018 at 10:52 AM, Rong Rong <[hidden email]> wrote:
>
> > Hi everyone,
> >
> > I was wondering if it is a good idea to support some external catalog
> > software, such as Apache HCatalog[2], out-of-the-box for the FLIP-24
> > SQL-Client[1]. There are many widely used catalogs that we can
> incorporate.
> > This way users won't have to always extend and create their own
> > ExternalCatalog.class separately and this could potentially make the
> > configuration part easier for SQL users.
> >
> > Thanks,
> > Rong
> >
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> 24+-+SQL+Client
> > [2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
> >
>



--
"So you have to trust that the dots will somehow connect in your future."
Reply | Threaded
Open this post in threaded view
|

Re: Support for out-of-the-box external catalog for SQL Client

Fabian Hueske-2
Hi everybody,

An HCatalog integration with the Table API/SQL would be great and be
helpful for many users!

A big +1 to that.

Thank you,
Fabian

Shuyi Chen <[hidden email]> schrieb am Mi., 11. Apr. 2018, 14:36:

> Thanks a lot, Rong and Peter.
>
> AFAIK, there is a flink hcatalog connector introduced in FLINK-1466
> <https://issues.apache.org/jira/browse/FLINK-1466> that is added by
> Fabian.
> And there is another JIRA in FLINK-1913
> <https://issues.apache.org/jira/browse/FLINK-1913> to document the use of
> connector.
>
> I think we can start with looking at the existing hcatalog  connector,
> adding missing documentation, and come up with a proposal to evolve the
> Flink HCatalog integration with ExternalCatalog, and the SQL client to make
> it both useful both SQL and non-SQL scenarios.
>
> Given we already have the integration implemented in AthenaX
> <https://github.com/uber/AthenaX> internally, we can help drive and
> contribute back to the community.
>
> Shuyi
>
> On Wed, Apr 11, 2018 at 12:01 PM, Peter Huang <[hidden email]>
> wrote:
>
> > Hi Rong,
> >
> > It is a good point out. I aligned with Fabian yesterday. It is a good
> work
> > that I can involve
> > to contribute back to Apache Flink after having AthenaX backfill support
> > internally.
> >
> >
> > Best Regards
> > Peter Huang
> >
> > On Wed, Apr 11, 2018 at 10:52 AM, Rong Rong <[hidden email]> wrote:
> >
> > > Hi everyone,
> > >
> > > I was wondering if it is a good idea to support some external catalog
> > > software, such as Apache HCatalog[2], out-of-the-box for the FLIP-24
> > > SQL-Client[1]. There are many widely used catalogs that we can
> > incorporate.
> > > This way users won't have to always extend and create their own
> > > ExternalCatalog.class separately and this could potentially make the
> > > configuration part easier for SQL users.
> > >
> > > Thanks,
> > > Rong
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > 24+-+SQL+Client
> > > [2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
> > >
> >
>
>
>
> --
> "So you have to trust that the dots will somehow connect in your future."
>
Reply | Threaded
Open this post in threaded view
|

Re: Support for out-of-the-box external catalog for SQL Client

Shuyi Chen
I've created master JIRA (https://issues.apache.org/jira/browse/FLINK-9171),
and included all HCatalog related JIRAs as subtasks. This make it easier to
track all HCatalog related effort in Flink. Thanks.

Shuyi

On Fri, Apr 13, 2018 at 12:36 PM, Fabian Hueske <[hidden email]> wrote:

> Hi everybody,
>
> An HCatalog integration with the Table API/SQL would be great and be
> helpful for many users!
>
> A big +1 to that.
>
> Thank you,
> Fabian
>
> Shuyi Chen <[hidden email]> schrieb am Mi., 11. Apr. 2018, 14:36:
>
> > Thanks a lot, Rong and Peter.
> >
> > AFAIK, there is a flink hcatalog connector introduced in FLINK-1466
> > <https://issues.apache.org/jira/browse/FLINK-1466> that is added by
> > Fabian.
> > And there is another JIRA in FLINK-1913
> > <https://issues.apache.org/jira/browse/FLINK-1913> to document the use
> of
> > connector.
> >
> > I think we can start with looking at the existing hcatalog  connector,
> > adding missing documentation, and come up with a proposal to evolve the
> > Flink HCatalog integration with ExternalCatalog, and the SQL client to
> make
> > it both useful both SQL and non-SQL scenarios.
> >
> > Given we already have the integration implemented in AthenaX
> > <https://github.com/uber/AthenaX> internally, we can help drive and
> > contribute back to the community.
> >
> > Shuyi
> >
> > On Wed, Apr 11, 2018 at 12:01 PM, Peter Huang <
> [hidden email]>
> > wrote:
> >
> > > Hi Rong,
> > >
> > > It is a good point out. I aligned with Fabian yesterday. It is a good
> > work
> > > that I can involve
> > > to contribute back to Apache Flink after having AthenaX backfill
> support
> > > internally.
> > >
> > >
> > > Best Regards
> > > Peter Huang
> > >
> > > On Wed, Apr 11, 2018 at 10:52 AM, Rong Rong <[hidden email]>
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I was wondering if it is a good idea to support some external catalog
> > > > software, such as Apache HCatalog[2], out-of-the-box for the FLIP-24
> > > > SQL-Client[1]. There are many widely used catalogs that we can
> > > incorporate.
> > > > This way users won't have to always extend and create their own
> > > > ExternalCatalog.class separately and this could potentially make the
> > > > configuration part easier for SQL users.
> > > >
> > > > Thanks,
> > > > Rong
> > > >
> > > >
> > > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > > 24+-+SQL+Client
> > > > [2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
> > > >
> > >
> >
> >
> >
> > --
> > "So you have to trust that the dots will somehow connect in your future."
> >
>



--
"So you have to trust that the dots will somehow connect in your future."
Reply | Threaded
Open this post in threaded view
|

Re: Support for out-of-the-box external catalog for SQL Client

Rong Rong
Thanks Peter, Fabian & Shuyi for the input.

I have also created a task https://issues.apache.org/jira/browse/FLINK-9172.
Then we can have external catalog factory support on SQL-Client that works
with the abstractions in Table/SQL API.

--
Rong


On Fri, Apr 13, 2018 at 5:00 PM, Shuyi Chen <[hidden email]> wrote:

> I've created master JIRA (https://issues.apache.org/jira/browse/FLINK-9171
> ),
> and included all HCatalog related JIRAs as subtasks. This make it easier to
> track all HCatalog related effort in Flink. Thanks.
>
> Shuyi
>
> On Fri, Apr 13, 2018 at 12:36 PM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi everybody,
> >
> > An HCatalog integration with the Table API/SQL would be great and be
> > helpful for many users!
> >
> > A big +1 to that.
> >
> > Thank you,
> > Fabian
> >
> > Shuyi Chen <[hidden email]> schrieb am Mi., 11. Apr. 2018, 14:36:
> >
> > > Thanks a lot, Rong and Peter.
> > >
> > > AFAIK, there is a flink hcatalog connector introduced in FLINK-1466
> > > <https://issues.apache.org/jira/browse/FLINK-1466> that is added by
> > > Fabian.
> > > And there is another JIRA in FLINK-1913
> > > <https://issues.apache.org/jira/browse/FLINK-1913> to document the use
> > of
> > > connector.
> > >
> > > I think we can start with looking at the existing hcatalog  connector,
> > > adding missing documentation, and come up with a proposal to evolve the
> > > Flink HCatalog integration with ExternalCatalog, and the SQL client to
> > make
> > > it both useful both SQL and non-SQL scenarios.
> > >
> > > Given we already have the integration implemented in AthenaX
> > > <https://github.com/uber/AthenaX> internally, we can help drive and
> > > contribute back to the community.
> > >
> > > Shuyi
> > >
> > > On Wed, Apr 11, 2018 at 12:01 PM, Peter Huang <
> > [hidden email]>
> > > wrote:
> > >
> > > > Hi Rong,
> > > >
> > > > It is a good point out. I aligned with Fabian yesterday. It is a good
> > > work
> > > > that I can involve
> > > > to contribute back to Apache Flink after having AthenaX backfill
> > support
> > > > internally.
> > > >
> > > >
> > > > Best Regards
> > > > Peter Huang
> > > >
> > > > On Wed, Apr 11, 2018 at 10:52 AM, Rong Rong <[hidden email]>
> > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I was wondering if it is a good idea to support some external
> catalog
> > > > > software, such as Apache HCatalog[2], out-of-the-box for the
> FLIP-24
> > > > > SQL-Client[1]. There are many widely used catalogs that we can
> > > > incorporate.
> > > > > This way users won't have to always extend and create their own
> > > > > ExternalCatalog.class separately and this could potentially make
> the
> > > > > configuration part easier for SQL users.
> > > > >
> > > > > Thanks,
> > > > > Rong
> > > > >
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > > > 24+-+SQL+Client
> > > > > [2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > "So you have to trust that the dots will somehow connect in your
> future."
> > >
> >
>
>
>
> --
> "So you have to trust that the dots will somehow connect in your future."
>
Reply | Threaded
Open this post in threaded view
|

Re: Support for out-of-the-box external catalog for SQL Client

Fabian Hueske-2
Great, Thank you Shuyi and Rong!

2018-04-14 3:03 GMT+02:00 Rong Rong <[hidden email]>:

> Thanks Peter, Fabian & Shuyi for the input.
>
> I have also created a task https://issues.apache.org/
> jira/browse/FLINK-9172.
> Then we can have external catalog factory support on SQL-Client that works
> with the abstractions in Table/SQL API.
>
> --
> Rong
>
>
> On Fri, Apr 13, 2018 at 5:00 PM, Shuyi Chen <[hidden email]> wrote:
>
> > I've created master JIRA (https://issues.apache.org/
> jira/browse/FLINK-9171
> > ),
> > and included all HCatalog related JIRAs as subtasks. This make it easier
> to
> > track all HCatalog related effort in Flink. Thanks.
> >
> > Shuyi
> >
> > On Fri, Apr 13, 2018 at 12:36 PM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi everybody,
> > >
> > > An HCatalog integration with the Table API/SQL would be great and be
> > > helpful for many users!
> > >
> > > A big +1 to that.
> > >
> > > Thank you,
> > > Fabian
> > >
> > > Shuyi Chen <[hidden email]> schrieb am Mi., 11. Apr. 2018, 14:36:
> > >
> > > > Thanks a lot, Rong and Peter.
> > > >
> > > > AFAIK, there is a flink hcatalog connector introduced in FLINK-1466
> > > > <https://issues.apache.org/jira/browse/FLINK-1466> that is added by
> > > > Fabian.
> > > > And there is another JIRA in FLINK-1913
> > > > <https://issues.apache.org/jira/browse/FLINK-1913> to document the
> use
> > > of
> > > > connector.
> > > >
> > > > I think we can start with looking at the existing hcatalog
> connector,
> > > > adding missing documentation, and come up with a proposal to evolve
> the
> > > > Flink HCatalog integration with ExternalCatalog, and the SQL client
> to
> > > make
> > > > it both useful both SQL and non-SQL scenarios.
> > > >
> > > > Given we already have the integration implemented in AthenaX
> > > > <https://github.com/uber/AthenaX> internally, we can help drive and
> > > > contribute back to the community.
> > > >
> > > > Shuyi
> > > >
> > > > On Wed, Apr 11, 2018 at 12:01 PM, Peter Huang <
> > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Rong,
> > > > >
> > > > > It is a good point out. I aligned with Fabian yesterday. It is a
> good
> > > > work
> > > > > that I can involve
> > > > > to contribute back to Apache Flink after having AthenaX backfill
> > > support
> > > > > internally.
> > > > >
> > > > >
> > > > > Best Regards
> > > > > Peter Huang
> > > > >
> > > > > On Wed, Apr 11, 2018 at 10:52 AM, Rong Rong <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I was wondering if it is a good idea to support some external
> > catalog
> > > > > > software, such as Apache HCatalog[2], out-of-the-box for the
> > FLIP-24
> > > > > > SQL-Client[1]. There are many widely used catalogs that we can
> > > > > incorporate.
> > > > > > This way users won't have to always extend and create their own
> > > > > > ExternalCatalog.class separately and this could potentially make
> > the
> > > > > > configuration part easier for SQL users.
> > > > > >
> > > > > > Thanks,
> > > > > > Rong
> > > > > >
> > > > > >
> > > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-
> > > > > 24+-+SQL+Client
> > > > > > [2] https://cwiki.apache.org/confluence/display/Hive/HCatalog
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > "So you have to trust that the dots will somehow connect in your
> > future."
> > > >
> > >
> >
> >
> >
> > --
> > "So you have to trust that the dots will somehow connect in your future."
> >
>