Hi all,
I apologize for sending the email to both accounts, but not sure where this topic fits better. In my team, we have been working in some PoCs and PoVs about new data architectures. As part of this work, we have implemented a library to connect Kudu and Flink. The library allows reading/writing from/to Kudu tablets using DataSet API and also writing to Kudu using DataStream API. You can find the code and documentation (including some examples) in https://github.com/rubencasado/Flink-Kudu Any comment/suggestion/contribution is very welcomed ☺ We will try to publish this contribution to the Apache Bahir project. Best ---------------------------------------- Rubén Casado Tejedor, PhD > accenture digital Big Data Manager ' + 34 629 009 429 • [hidden email]<mailto:[hidden email]> ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com |
Hi Ruben,
thanks for sharing this! A Flink Kudu connector is a great contribution and Bahir seems to be the right place for it. Thanks, Fabian 2017-03-27 15:35 GMT+02:00 <[hidden email]>: > Hi all, > > I apologize for sending the email to both accounts, but not sure where > this topic fits better. > > In my team, we have been working in some PoCs and PoVs about new data > architectures. As part of this work, we have implemented a library to > connect Kudu and Flink. The library allows reading/writing from/to Kudu > tablets using DataSet API and also writing to Kudu using DataStream API. > > You can find the code and documentation (including some examples) in > https://github.com/rubencasado/Flink-Kudu > > Any comment/suggestion/contribution is very welcomed ☺ > > We will try to publish this contribution to the Apache Bahir project. > > Best > > ---------------------------------------- > Rubén Casado Tejedor, PhD > > accenture digital > Big Data Manager > ' + 34 629 009 429 > • [hidden email]<mailto:ruben. > [hidden email]> > > ________________________________ > > This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. Where allowed > by local law, electronic communications with Accenture and its affiliates, > including e-mail and instant messaging (including content), may be scanned > by our systems for the purposes of information security and assessment of > internal compliance with Accenture policy. > ____________________________________________________________ > __________________________ > > www.accenture.com > |
Hi, Fabian
Do we have plan to replace Flink connectors with bahir-flink[1]? [1] https://github.com/apache/Bahir-flink 在 2017/3/28 上午12:15, "Fabian Hueske" <[hidden email]> 写入: >Hi Ruben, > >thanks for sharing this! >A Flink Kudu connector is a great contribution and Bahir seems to be the >right place for it. > >Thanks, Fabian > > >2017-03-27 15:35 GMT+02:00 <[hidden email]>: > >> Hi all, >> >> I apologize for sending the email to both accounts, but not sure where >> this topic fits better. >> >> In my team, we have been working in some PoCs and PoVs about new data >> architectures. As part of this work, we have implemented a library to >> connect Kudu and Flink. The library allows reading/writing from/to Kudu >> tablets using DataSet API and also writing to Kudu using DataStream API. >> >> You can find the code and documentation (including some examples) in >> https://github.com/rubencasado/Flink-Kudu >> >> Any comment/suggestion/contribution is very welcomed ☺ >> >> We will try to publish this contribution to the Apache Bahir project. >> >> Best >> >> ---------------------------------------- >> Rubén Casado Tejedor, PhD >> > accenture digital >> Big Data Manager >> ' + 34 629 009 429 >> • [hidden email]<mailto:ruben. >> [hidden email]> >> >> ________________________________ >> >> This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise confidential information. If you >>have >> received it in error, please notify the sender immediately and delete >>the >> original. Any other use of the e-mail by you is prohibited. Where >>allowed >> by local law, electronic communications with Accenture and its >>affiliates, >> including e-mail and instant messaging (including content), may be >>scanned >> by our systems for the purposes of information security and assessment >>of >> internal compliance with Accenture policy. >> ____________________________________________________________ >> __________________________ >> >> www.accenture.com >> |
No, we do not want to move all connectors to Apache Bahir or replace the
connectors by Bahir. The Flink community aims to maintain the most important connectors within Flink. Maintaining all connectors would be a huge effort. So, we decided to move some of the less frequently used connectors to Bahir. Best, Fabian 2017-03-28 8:31 GMT+02:00 shijinkui <[hidden email]>: > Hi, Fabian > Do we have plan to replace Flink connectors with bahir-flink[1]? > > [1] https://github.com/apache/Bahir-flink > > 在 2017/3/28 上午12:15, "Fabian Hueske" <[hidden email]> 写入: > > >Hi Ruben, > > > >thanks for sharing this! > >A Flink Kudu connector is a great contribution and Bahir seems to be the > >right place for it. > > > >Thanks, Fabian > > > > > >2017-03-27 15:35 GMT+02:00 <[hidden email]>: > > > >> Hi all, > >> > >> I apologize for sending the email to both accounts, but not sure where > >> this topic fits better. > >> > >> In my team, we have been working in some PoCs and PoVs about new data > >> architectures. As part of this work, we have implemented a library to > >> connect Kudu and Flink. The library allows reading/writing from/to Kudu > >> tablets using DataSet API and also writing to Kudu using DataStream API. > >> > >> You can find the code and documentation (including some examples) in > >> https://github.com/rubencasado/Flink-Kudu > >> > >> Any comment/suggestion/contribution is very welcomed ☺ > >> > >> We will try to publish this contribution to the Apache Bahir project. > >> > >> Best > >> > >> ---------------------------------------- > >> Rubén Casado Tejedor, PhD > >> > accenture digital > >> Big Data Manager > >> ' + 34 629 009 429 > >> • [hidden email]<mailto:ruben. > >> [hidden email]> > >> > >> ________________________________ > >> > >> This message is for the designated recipient only and may contain > >> privileged, proprietary, or otherwise confidential information. If you > >>have > >> received it in error, please notify the sender immediately and delete > >>the > >> original. Any other use of the e-mail by you is prohibited. Where > >>allowed > >> by local law, electronic communications with Accenture and its > >>affiliates, > >> including e-mail and instant messaging (including content), may be > >>scanned > >> by our systems for the purposes of information security and assessment > >>of > >> internal compliance with Accenture policy. > >> ____________________________________________________________ > >> __________________________ > >> > >> www.accenture.com > >> > > |
We are currently looking into how we can keep the size of the code base
under control (because it is growing super large). Part is moving libraries into a dedicated subrepository (there is a separate mailing list thread on that) and some connectors to Bahir. Connectors can move between Flink and Bahir, for example it makes sense to move heavily worked on connectors to the Flink code base. On Tue, Mar 28, 2017 at 10:01 AM, Fabian Hueske <[hidden email]> wrote: > No, we do not want to move all connectors to Apache Bahir or replace the > connectors by Bahir. > > The Flink community aims to maintain the most important connectors within > Flink. > Maintaining all connectors would be a huge effort. So, we decided to move > some of the less frequently used connectors to Bahir. > > Best, Fabian > > > > 2017-03-28 8:31 GMT+02:00 shijinkui <[hidden email]>: > >> Hi, Fabian >> Do we have plan to replace Flink connectors with bahir-flink[1]? >> >> [1] https://github.com/apache/Bahir-flink >> >> 在 2017/3/28 上午12:15, "Fabian Hueske" <[hidden email]> 写入: >> >> >Hi Ruben, >> > >> >thanks for sharing this! >> >A Flink Kudu connector is a great contribution and Bahir seems to be the >> >right place for it. >> > >> >Thanks, Fabian >> > >> > >> >2017-03-27 15:35 GMT+02:00 <[hidden email]>: >> > >> >> Hi all, >> >> >> >> I apologize for sending the email to both accounts, but not sure where >> >> this topic fits better. >> >> >> >> In my team, we have been working in some PoCs and PoVs about new data >> >> architectures. As part of this work, we have implemented a library to >> >> connect Kudu and Flink. The library allows reading/writing from/to Kudu >> >> tablets using DataSet API and also writing to Kudu using DataStream >> API. >> >> >> >> You can find the code and documentation (including some examples) in >> >> https://github.com/rubencasado/Flink-Kudu >> >> >> >> Any comment/suggestion/contribution is very welcomed ☺ >> >> >> >> We will try to publish this contribution to the Apache Bahir project. >> >> >> >> Best >> >> >> >> ---------------------------------------- >> >> Rubén Casado Tejedor, PhD >> >> > accenture digital >> >> Big Data Manager >> >> ' + 34 629 009 429 >> >> • [hidden email]<mailto:ruben. >> >> [hidden email]> >> >> >> >> ________________________________ >> >> >> >> This message is for the designated recipient only and may contain >> >> privileged, proprietary, or otherwise confidential information. If you >> >>have >> >> received it in error, please notify the sender immediately and delete >> >>the >> >> original. Any other use of the e-mail by you is prohibited. Where >> >>allowed >> >> by local law, electronic communications with Accenture and its >> >>affiliates, >> >> including e-mail and instant messaging (including content), may be >> >>scanned >> >> by our systems for the purposes of information security and assessment >> >>of >> >> internal compliance with Accenture policy. >> >> ____________________________________________________________ >> >> __________________________ >> >> >> >> www.accenture.com >> >> >> >> > |
This make sense. Only one problem, bahir-flink is not very active. Maybe need more committee to pay some time on it, and have some new plan.
We also plan to support carbondata[1] in bahir-flink. By the way, to keep the code base thin, can rewrite flink with Scala step by step until Flink 2.0 :))) [1] https://github.com/apache/incubator-carbondata <https://github.com/apache/incubator-carbondata> > 在 2017年3月28日,下午5:46,Stephan Ewen <[hidden email]> 写道: > > We are currently looking into how we can keep the size of the code base > under control (because it is growing super large). > > Part is moving libraries into a dedicated subrepository (there is a > separate mailing list thread on that) and some connectors to Bahir. > Connectors can move between Flink and Bahir, for example it makes sense to > move heavily worked on connectors to the Flink code base. > > On Tue, Mar 28, 2017 at 10:01 AM, Fabian Hueske <[hidden email]> wrote: > >> No, we do not want to move all connectors to Apache Bahir or replace the >> connectors by Bahir. >> >> The Flink community aims to maintain the most important connectors within >> Flink. >> Maintaining all connectors would be a huge effort. So, we decided to move >> some of the less frequently used connectors to Bahir. >> >> Best, Fabian >> >> >> >> 2017-03-28 8:31 GMT+02:00 shijinkui <[hidden email]>: >> >>> Hi, Fabian >>> Do we have plan to replace Flink connectors with bahir-flink[1]? >>> >>> [1] https://github.com/apache/Bahir-flink >>> >>> 在 2017/3/28 上午12:15, "Fabian Hueske" <[hidden email]> 写入: >>> >>>> Hi Ruben, >>>> >>>> thanks for sharing this! >>>> A Flink Kudu connector is a great contribution and Bahir seems to be the >>>> right place for it. >>>> >>>> Thanks, Fabian >>>> >>>> >>>> 2017-03-27 15:35 GMT+02:00 <[hidden email]>: >>>> >>>>> Hi all, >>>>> >>>>> I apologize for sending the email to both accounts, but not sure where >>>>> this topic fits better. >>>>> >>>>> In my team, we have been working in some PoCs and PoVs about new data >>>>> architectures. As part of this work, we have implemented a library to >>>>> connect Kudu and Flink. The library allows reading/writing from/to Kudu >>>>> tablets using DataSet API and also writing to Kudu using DataStream >>> API. >>>>> >>>>> You can find the code and documentation (including some examples) in >>>>> https://github.com/rubencasado/Flink-Kudu >>>>> >>>>> Any comment/suggestion/contribution is very welcomed ☺ >>>>> >>>>> We will try to publish this contribution to the Apache Bahir project. >>>>> >>>>> Best >>>>> >>>>> ---------------------------------------- >>>>> Rubén Casado Tejedor, PhD >>>>>> accenture digital >>>>> Big Data Manager >>>>> ' + 34 629 009 429 >>>>> • [hidden email]<mailto:ruben. >>>>> [hidden email]> >>>>> >>>>> ________________________________ >>>>> >>>>> This message is for the designated recipient only and may contain >>>>> privileged, proprietary, or otherwise confidential information. If you >>>>> have >>>>> received it in error, please notify the sender immediately and delete >>>>> the >>>>> original. Any other use of the e-mail by you is prohibited. Where >>>>> allowed >>>>> by local law, electronic communications with Accenture and its >>>>> affiliates, >>>>> including e-mail and instant messaging (including content), may be >>>>> scanned >>>>> by our systems for the purposes of information security and assessment >>>>> of >>>>> internal compliance with Accenture policy. >>>>> ____________________________________________________________ >>>>> __________________________ >>>>> >>>>> www.accenture.com >>>>> >>> >>> >> |
Free forum by Nabble | Edit this page |