Hi, all
1. In order to support end-to-end pushed source, I create FLINK-4630<https://issues.apache.org/jira/browse/FLINK-4630>. I want to know whether is this idea worth? ------------------- When source stream get start, listen a provided tcp port, receive stream data from user data source. This netty tcp source is keepping alive and end-to-end, that is from business system to flink worker directly. user app push -----> netty server source of Flink describe the source in detail below: 1. source run as a netty tcp server 2. user provide a tcp port, if the port is in used, increace the port number between 1024 to 65535. Source can parallel. 3. callback the provided url to report the real port to listen 4. user push streaming data to netty server, then collect the data to flink Thanks Jinkui Shi |
I think that could be an interesting source. Two quick questions to move
forward - To keep the Flink code base from becoming too big (hard to maintain and test) we started working with Apache Bahir as a project dedicated to streaming connectors. Would that be a good target for the connector? - What are your thoughts on fault tolerance for that connector? On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> wrote: > Hi, all > > 1. In order to support end-to-end pushed source, I create FLINK-4630< > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know whether > is this idea worth? > > ------------------- > When source stream get start, listen a provided tcp port, receive stream > data from user data source. > This netty tcp source is keepping alive and end-to-end, that is from > business system to flink worker directly. > > user app push -----> netty server source of Flink > > describe the source in detail below: > > 1. source run as a netty tcp server > 2. user provide a tcp port, if the port is in used, increace the port > number between 1024 to 65535. Source can parallel. > 3. callback the provided url to report the real port to listen > 4. user push streaming data to netty server, then collect the data to > flink > > > Thanks > > Jinkui Shi > > |
Hey, Stephan Ewen
1. bahir's target is spark. The contributer are rxin, srowen, tdas, mateiz and so on. If we want bahir used by flink, we can suggest bahir provide streaming connecter interface, such as store(), start(), stop(), restart(), receiving(Any)... Then same streaming connector can be implemented by spark and flink. But I think this is impossible, as bahir depend spark-streaming and spark sql. 2. About connector fault tolerance. Bahir's mqtt and akka connector are themselves' storage. But netty have no persist data feature. I think we can append data to a ringbuffer. When SourceContext collect() throw error, then write message to inform client to stop send message. When flink SourceContext is normal, then write the ringbuffer data to flink, inform client to go on. Because pushing mode is hard to control the flow throughput, the upstream client can. This netty connector's purpose is end-to-end streaming, minimum time delay. 3. Later on, We can provide http protocol ability, put, post, get, even websocket or jersey restful based on netty. -----邮件原件----- 发件人: Stephan Ewen [mailto:[hidden email]] 发送时间: 2016年9月27日 16:54 收件人: [hidden email] 主题: Re: [DISCUSS] add netty tcp/restful pushed source support I think that could be an interesting source. Two quick questions to move forward - To keep the Flink code base from becoming too big (hard to maintain and test) we started working with Apache Bahir as a project dedicated to streaming connectors. Would that be a good target for the connector? - What are your thoughts on fault tolerance for that connector? On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> wrote: > Hi, all > > 1. In order to support end-to-end pushed source, I create FLINK-4630< > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > whether is this idea worth? > > ------------------- > When source stream get start, listen a provided tcp port, receive > stream data from user data source. > This netty tcp source is keepping alive and end-to-end, that is from > business system to flink worker directly. > > user app push -----> netty server source of Flink > > describe the source in detail below: > > 1. source run as a netty tcp server > 2. user provide a tcp port, if the port is in used, increace the port > number between 1024 to 65535. Source can parallel. > 3. callback the provided url to report the real port to listen > 4. user push streaming data to netty server, then collect the data to > flink > > > Thanks > > Jinkui Shi > > |
Apache Bahir's website only suggests support for additional frameworks, but
there is a Flink repository at https://github.com/apache/bahir-flink On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <[hidden email]> wrote: > Hey, Stephan Ewen > > 1. bahir's target is spark. The contributer are rxin, srowen, tdas, > mateiz and so on. > If we want bahir used by flink, we can suggest bahir provide > streaming connecter interface, such as store(), start(), stop(), restart(), > receiving(Any)... > Then same streaming connector can be implemented by spark and > flink. But I think this is impossible, as bahir depend spark-streaming and > spark sql. > 2. About connector fault tolerance. Bahir's mqtt and akka connector > are themselves' storage. But netty have no persist data feature. > I think we can append data to a ringbuffer. When SourceContext > collect() throw error, then write message to inform client to stop send > message. When flink SourceContext is normal, then write the ringbuffer data > to flink, inform client to go on. > Because pushing mode is hard to control the flow throughput, the > upstream client can. > This netty connector's purpose is end-to-end streaming, minimum > time delay. > 3. Later on, We can provide http protocol ability, put, post, get, > even websocket or jersey restful based on netty. > > -----邮件原件----- > 发件人: Stephan Ewen [mailto:[hidden email]] > 发送时间: 2016年9月27日 16:54 > 收件人: [hidden email] > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > I think that could be an interesting source. Two quick questions to move > forward > > - To keep the Flink code base from becoming too big (hard to maintain and > test) we started working with Apache Bahir as a project dedicated to > streaming connectors. Would that be a good target for the connector? > > - What are your thoughts on fault tolerance for that connector? > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> wrote: > > > Hi, all > > > > 1. In order to support end-to-end pushed source, I create FLINK-4630< > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > whether is this idea worth? > > > > ------------------- > > When source stream get start, listen a provided tcp port, receive > > stream data from user data source. > > This netty tcp source is keepping alive and end-to-end, that is from > > business system to flink worker directly. > > > > user app push -----> netty server source of Flink > > > > describe the source in detail below: > > > > 1. source run as a netty tcp server > > 2. user provide a tcp port, if the port is in used, increace the port > > number between 1024 to 65535. Source can parallel. > > 3. callback the provided url to report the real port to listen > > 4. user push streaming data to netty server, then collect the data to > > flink > > > > > > Thanks > > > > Jinkui Shi > > > > > |
It's nice. Will present flink source connector be pushed to bahir-flink?
I can add netty-source to bahir-flink. Maven repository have no bahir-flink's. https://mvnrepository.com/artifact/org.apache.bahir -----邮件原件----- 发件人: Greg Hogan [mailto:[hidden email]] 发送时间: 2016年9月27日 20:58 收件人: [hidden email] 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support Apache Bahir's website only suggests support for additional frameworks, but there is a Flink repository at https://github.com/apache/bahir-flink On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <[hidden email]> wrote: > Hey, Stephan Ewen > > 1. bahir's target is spark. The contributer are rxin, srowen, tdas, > mateiz and so on. > If we want bahir used by flink, we can suggest bahir provide > streaming connecter interface, such as store(), start(), stop(), > restart(), receiving(Any)... > Then same streaming connector can be implemented by spark and > flink. But I think this is impossible, as bahir depend spark-streaming > and spark sql. > 2. About connector fault tolerance. Bahir's mqtt and akka connector > are themselves' storage. But netty have no persist data feature. > I think we can append data to a ringbuffer. When SourceContext > collect() throw error, then write message to inform client to stop > send message. When flink SourceContext is normal, then write the > ringbuffer data to flink, inform client to go on. > Because pushing mode is hard to control the flow throughput, > the upstream client can. > This netty connector's purpose is end-to-end streaming, > minimum time delay. > 3. Later on, We can provide http protocol ability, put, post, get, > even websocket or jersey restful based on netty. > > -----邮件原件----- > 发件人: Stephan Ewen [mailto:[hidden email]] > 发送时间: 2016年9月27日 16:54 > 收件人: [hidden email] > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > I think that could be an interesting source. Two quick questions to > move forward > > - To keep the Flink code base from becoming too big (hard to > maintain and > test) we started working with Apache Bahir as a project dedicated to > streaming connectors. Would that be a good target for the connector? > > - What are your thoughts on fault tolerance for that connector? > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> wrote: > > > Hi, all > > > > 1. In order to support end-to-end pushed source, I create FLINK-4630< > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > whether is this idea worth? > > > > ------------------- > > When source stream get start, listen a provided tcp port, receive > > stream data from user data source. > > This netty tcp source is keepping alive and end-to-end, that is from > > business system to flink worker directly. > > > > user app push -----> netty server source of Flink > > > > describe the source in detail below: > > > > 1. source run as a netty tcp server > > 2. user provide a tcp port, if the port is in used, increace the port > > number between 1024 to 65535. Source can parallel. > > 3. callback the provided url to report the real port to listen > > 4. user push streaming data to netty server, then collect the data to > > flink > > > > > > Thanks > > > > Jinkui Shi > > > > > |
The Bahir-Flink stuff is fairly new - the first release has not yet
happened. Robert is pushing that, will probably happen once he is back. On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <[hidden email]> wrote: > It's nice. Will present flink source connector be pushed to bahir-flink? > I can add netty-source to bahir-flink. > > Maven repository have no bahir-flink's. > https://mvnrepository.com/artifact/org.apache.bahir > > -----邮件原件----- > 发件人: Greg Hogan [mailto:[hidden email]] > 发送时间: 2016年9月27日 20:58 > 收件人: [hidden email] > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support > > Apache Bahir's website only suggests support for additional frameworks, > but there is a Flink repository at > https://github.com/apache/bahir-flink > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <[hidden email]> wrote: > > > Hey, Stephan Ewen > > > > 1. bahir's target is spark. The contributer are rxin, srowen, tdas, > > mateiz and so on. > > If we want bahir used by flink, we can suggest bahir provide > > streaming connecter interface, such as store(), start(), stop(), > > restart(), receiving(Any)... > > Then same streaming connector can be implemented by spark and > > flink. But I think this is impossible, as bahir depend spark-streaming > > and spark sql. > > 2. About connector fault tolerance. Bahir's mqtt and akka connector > > are themselves' storage. But netty have no persist data feature. > > I think we can append data to a ringbuffer. When SourceContext > > collect() throw error, then write message to inform client to stop > > send message. When flink SourceContext is normal, then write the > > ringbuffer data to flink, inform client to go on. > > Because pushing mode is hard to control the flow throughput, > > the upstream client can. > > This netty connector's purpose is end-to-end streaming, > > minimum time delay. > > 3. Later on, We can provide http protocol ability, put, post, get, > > even websocket or jersey restful based on netty. > > > > -----邮件原件----- > > 发件人: Stephan Ewen [mailto:[hidden email]] > > 发送时间: 2016年9月27日 16:54 > > 收件人: [hidden email] > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > > > I think that could be an interesting source. Two quick questions to > > move forward > > > > - To keep the Flink code base from becoming too big (hard to > > maintain and > > test) we started working with Apache Bahir as a project dedicated to > > streaming connectors. Would that be a good target for the connector? > > > > - What are your thoughts on fault tolerance for that connector? > > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> wrote: > > > > > Hi, all > > > > > > 1. In order to support end-to-end pushed source, I create > FLINK-4630< > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > > whether is this idea worth? > > > > > > ------------------- > > > When source stream get start, listen a provided tcp port, receive > > > stream data from user data source. > > > This netty tcp source is keepping alive and end-to-end, that is from > > > business system to flink worker directly. > > > > > > user app push -----> netty server source of Flink > > > > > > describe the source in detail below: > > > > > > 1. source run as a netty tcp server > > > 2. user provide a tcp port, if the port is in used, increace the > port > > > number between 1024 to 65535. Source can parallel. > > > 3. callback the provided url to report the real port to listen > > > 4. user push streaming data to netty server, then collect the data > to > > > flink > > > > > > > > > Thanks > > > > > > Jinkui Shi > > > > > > > > > |
Just a quick update on this one: The bahir community started already
discussing the first bahir-flink release. I expect it to happen soon. I would really like to see the netty source in Bahir. On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen <[hidden email]> wrote: > The Bahir-Flink stuff is fairly new - the first release has not yet > happened. > Robert is pushing that, will probably happen once he is back. > > On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <[hidden email]> wrote: > > > It's nice. Will present flink source connector be pushed to bahir-flink? > > I can add netty-source to bahir-flink. > > > > Maven repository have no bahir-flink's. > > https://mvnrepository.com/artifact/org.apache.bahir > > > > -----邮件原件----- > > 发件人: Greg Hogan [mailto:[hidden email]] > > 发送时间: 2016年9月27日 20:58 > > 收件人: [hidden email] > > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support > > > > Apache Bahir's website only suggests support for additional frameworks, > > but there is a Flink repository at > > https://github.com/apache/bahir-flink > > > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <[hidden email]> wrote: > > > > > Hey, Stephan Ewen > > > > > > 1. bahir's target is spark. The contributer are rxin, srowen, > tdas, > > > mateiz and so on. > > > If we want bahir used by flink, we can suggest bahir provide > > > streaming connecter interface, such as store(), start(), stop(), > > > restart(), receiving(Any)... > > > Then same streaming connector can be implemented by spark and > > > flink. But I think this is impossible, as bahir depend spark-streaming > > > and spark sql. > > > 2. About connector fault tolerance. Bahir's mqtt and akka > connector > > > are themselves' storage. But netty have no persist data feature. > > > I think we can append data to a ringbuffer. When SourceContext > > > collect() throw error, then write message to inform client to stop > > > send message. When flink SourceContext is normal, then write the > > > ringbuffer data to flink, inform client to go on. > > > Because pushing mode is hard to control the flow throughput, > > > the upstream client can. > > > This netty connector's purpose is end-to-end streaming, > > > minimum time delay. > > > 3. Later on, We can provide http protocol ability, put, post, get, > > > even websocket or jersey restful based on netty. > > > > > > -----邮件原件----- > > > 发件人: Stephan Ewen [mailto:[hidden email]] > > > 发送时间: 2016年9月27日 16:54 > > > 收件人: [hidden email] > > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > > > > > I think that could be an interesting source. Two quick questions to > > > move forward > > > > > > - To keep the Flink code base from becoming too big (hard to > > > maintain and > > > test) we started working with Apache Bahir as a project dedicated to > > > streaming connectors. Would that be a good target for the connector? > > > > > > - What are your thoughts on fault tolerance for that connector? > > > > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> > wrote: > > > > > > > Hi, all > > > > > > > > 1. In order to support end-to-end pushed source, I create > > FLINK-4630< > > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > > > whether is this idea worth? > > > > > > > > ------------------- > > > > When source stream get start, listen a provided tcp port, receive > > > > stream data from user data source. > > > > This netty tcp source is keepping alive and end-to-end, that is from > > > > business system to flink worker directly. > > > > > > > > user app push -----> netty server source of Flink > > > > > > > > describe the source in detail below: > > > > > > > > 1. source run as a netty tcp server > > > > 2. user provide a tcp port, if the port is in used, increace the > > port > > > > number between 1024 to 65535. Source can parallel. > > > > 3. callback the provided url to report the real port to listen > > > > 4. user push streaming data to netty server, then collect the data > > to > > > > flink > > > > > > > > > > > > Thanks > > > > > > > > Jinkui Shi > > > > > > > > > > > > > > |
+1 to add the source to Bahir
That is an easier way to iterate fast on this and release quickly On Wed, Oct 12, 2016 at 10:38 AM, Robert Metzger <[hidden email]> wrote: > Just a quick update on this one: The bahir community started already > discussing the first bahir-flink release. I expect it to happen soon. > > I would really like to see the netty source in Bahir. > > On Wed, Sep 28, 2016 at 3:18 PM, Stephan Ewen <[hidden email]> wrote: > > > The Bahir-Flink stuff is fairly new - the first release has not yet > > happened. > > Robert is pushing that, will probably happen once he is back. > > > > On Tue, Sep 27, 2016 at 3:31 PM, shijinkui <[hidden email]> wrote: > > > > > It's nice. Will present flink source connector be pushed to > bahir-flink? > > > I can add netty-source to bahir-flink. > > > > > > Maven repository have no bahir-flink's. > > > https://mvnrepository.com/artifact/org.apache.bahir > > > > > > -----邮件原件----- > > > 发件人: Greg Hogan [mailto:[hidden email]] > > > 发送时间: 2016年9月27日 20:58 > > > 收件人: [hidden email] > > > 主题: Re: 答复: [DISCUSS] add netty tcp/restful pushed source support > > > > > > Apache Bahir's website only suggests support for additional frameworks, > > > but there is a Flink repository at > > > https://github.com/apache/bahir-flink > > > > > > On Tue, Sep 27, 2016 at 8:38 AM, shijinkui <[hidden email]> > wrote: > > > > > > > Hey, Stephan Ewen > > > > > > > > 1. bahir's target is spark. The contributer are rxin, srowen, > > tdas, > > > > mateiz and so on. > > > > If we want bahir used by flink, we can suggest bahir provide > > > > streaming connecter interface, such as store(), start(), stop(), > > > > restart(), receiving(Any)... > > > > Then same streaming connector can be implemented by spark and > > > > flink. But I think this is impossible, as bahir depend > spark-streaming > > > > and spark sql. > > > > 2. About connector fault tolerance. Bahir's mqtt and akka > > connector > > > > are themselves' storage. But netty have no persist data feature. > > > > I think we can append data to a ringbuffer. When > SourceContext > > > > collect() throw error, then write message to inform client to stop > > > > send message. When flink SourceContext is normal, then write the > > > > ringbuffer data to flink, inform client to go on. > > > > Because pushing mode is hard to control the flow throughput, > > > > the upstream client can. > > > > This netty connector's purpose is end-to-end streaming, > > > > minimum time delay. > > > > 3. Later on, We can provide http protocol ability, put, post, > get, > > > > even websocket or jersey restful based on netty. > > > > > > > > -----邮件原件----- > > > > 发件人: Stephan Ewen [mailto:[hidden email]] > > > > 发送时间: 2016年9月27日 16:54 > > > > 收件人: [hidden email] > > > > 主题: Re: [DISCUSS] add netty tcp/restful pushed source support > > > > > > > > I think that could be an interesting source. Two quick questions to > > > > move forward > > > > > > > > - To keep the Flink code base from becoming too big (hard to > > > > maintain and > > > > test) we started working with Apache Bahir as a project dedicated to > > > > streaming connectors. Would that be a good target for the connector? > > > > > > > > - What are your thoughts on fault tolerance for that connector? > > > > > > > > On Mon, Sep 26, 2016 at 3:01 PM, shijinkui <[hidden email]> > > wrote: > > > > > > > > > Hi, all > > > > > > > > > > 1. In order to support end-to-end pushed source, I create > > > FLINK-4630< > > > > > https://issues.apache.org/jira/browse/FLINK-4630>. I want to know > > > > > whether is this idea worth? > > > > > > > > > > ------------------- > > > > > When source stream get start, listen a provided tcp port, receive > > > > > stream data from user data source. > > > > > This netty tcp source is keepping alive and end-to-end, that is > from > > > > > business system to flink worker directly. > > > > > > > > > > user app push -----> netty server source of Flink > > > > > > > > > > describe the source in detail below: > > > > > > > > > > 1. source run as a netty tcp server > > > > > 2. user provide a tcp port, if the port is in used, increace the > > > port > > > > > number between 1024 to 65535. Source can parallel. > > > > > 3. callback the provided url to report the real port to listen > > > > > 4. user push streaming data to netty server, then collect the > data > > > to > > > > > flink > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Jinkui Shi > > > > > > > > > > > > > > > > > > > > |
Free forum by Nabble | Edit this page |