Connecting Flink and Hive

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Connecting Flink and Hive

Alan Gates-2
I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1].   But I recall recently seeing a message on the dev list about moving some of the third party connectors out of Flink as devs were having problems maintaining them.  So, is this the sort of thing I should contribute to Flink or is the plan to house these types of things elsewhere?

Alan.

1. https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
Reply | Threaded
Open this post in threaded view
|

Re: Connecting Flink and Hive

Márton Balassi
Hi Alan,

Your contribution is more than welcome. It would be a great addition to
flink-streaming-connectors. At some point we might move some of these to a
"Flink Packages" repository, similarly to the Spark approach, but currently
the best place to have them is the internal connectors.

Robert in the previous mail was referring to some file system based
streaming sources, those are tricky to maintain as they have many
overloaded versions. Those will be moved to a different place within the
Flink repo.

Best,

Marton

On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote:

> I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1].
>  But I recall recently seeing a message on the dev list about moving some
> of the third party connectors out of Flink as devs were having problems
> maintaining them.  So, is this the sort of thing I should contribute to
> Flink or is the plan to house these types of things elsewhere?
>
> Alan.
>
> 1. https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
Reply | Threaded
Open this post in threaded view
|

Re: Connecting Flink and Hive

Robert Metzger
Hi Alan,

since you are a committer of Flink I don't see any issues merging such a
connector to Flink.
The reason why we didn't move the connectors out of the Flink repository
was the additional overhead for the CI system, Maven files, Apache release
process etc. so we decided to maintain connectors within Flink for now.

Please let us know if you have any questions while implementing the sink.


On Thu, Jul 7, 2016 at 11:17 AM, Márton Balassi <[hidden email]>
wrote:

> Hi Alan,
>
> Your contribution is more than welcome. It would be a great addition to
> flink-streaming-connectors. At some point we might move some of these to a
> "Flink Packages" repository, similarly to the Spark approach, but currently
> the best place to have them is the internal connectors.
>
> Robert in the previous mail was referring to some file system based
> streaming sources, those are tricky to maintain as they have many
> overloaded versions. Those will be moved to a different place within the
> Flink repo.
>
> Best,
>
> Marton
>
> On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote:
>
> > I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1].
> >  But I recall recently seeing a message on the dev list about moving some
> > of the third party connectors out of Flink as devs were having problems
> > maintaining them.  So, is this the sort of thing I should contribute to
> > Flink or is the plan to house these types of things elsewhere?
> >
> > Alan.
> >
> > 1.
> https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
>
Reply | Threaded
Open this post in threaded view
|

Re: Connecting Flink and Hive

Ufuk Celebi-2
In reply to this post by Alan Gates-2
Hey Alan,

as Marton said your contribution is more than welcome. :-)

The discussion around moving some contributions outside of the main
repository did never come to a final conclusion. Therefore, we
currently have most of the connectors inside of main Flink repo. As
long as there is no concrete proposal or guide line about what we keep
in Flink and what not, I think it's best to contribute it directly to
Flink.

– Ufuk


On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote:
> I’d like to work on creating a Flink Sink for Hive’s streaming ingest[1].   But I recall recently seeing a message on the dev list about moving some of the third party connectors out of Flink as devs were having problems maintaining them.  So, is this the sort of thing I should contribute to Flink or is the plan to house these types of things elsewhere?
>
> Alan.
>
> 1. https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
Reply | Threaded
Open this post in threaded view
|

Re: Connecting Flink and Hive

Vladislav Pernin
Moreover, that would reduce the build duration.
Le 7 juil. 2016 11:25, "Ufuk Celebi" <[hidden email]> a écrit :

> Hey Alan,
>
> as Marton said your contribution is more than welcome. :-)
>
> The discussion around moving some contributions outside of the main
> repository did never come to a final conclusion. Therefore, we
> currently have most of the connectors inside of main Flink repo. As
> long as there is no concrete proposal or guide line about what we keep
> in Flink and what not, I think it's best to contribute it directly to
> Flink.
>
> – Ufuk
>
>
> On Wed, Jul 6, 2016 at 5:14 PM, Alan Gates <[hidden email]> wrote:
> > I’d like to work on creating a Flink Sink for Hive’s streaming
> ingest[1].   But I recall recently seeing a message on the dev list about
> moving some of the third party connectors out of Flink as devs were having
> problems maintaining them.  So, is this the sort of thing I should
> contribute to Flink or is the plan to house these types of things elsewhere?
> >
> > Alan.
> >
> > 1.
> https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
>