Would like to add a connector to DistributedLog.

[Jia Zhai]

Hi,
I would like to add a connector to DistributedLog, which recently published
by twitter. :)
Is there any special instructions or best practise to add a new connector?
Any suggestion is very appreciated.

The JIRA tickets(FLINK-3987
<https://issues.apache.org/jira/browse/FLINK-3987> and FLINK-3988
<https://issues.apache.org/jira/browse/FLINK-3988>) has been added for this
work,

All the infomation of DistributedLog, could be found at here:
http://distributedlog.io.

Thanks a lot.
-Jia

[Jia Zhai]

+Distributedlog groups to get pertential help and instructions.
Thanks.

On Sun, May 29, 2016 at 11:33 AM, Jia Zhai <[hidden email]> wrote:

> Hi,
> I would like to add a connector to DistributedLog, which recently
> published by twitter. :)
> Is there any special instructions or best practise to add a new
> connector?  Any suggestion is very appreciated.
>
> The JIRA tickets(FLINK-3987
> <https://issues.apache.org/jira/browse/FLINK-3987> and FLINK-3988
> <https://issues.apache.org/jira/browse/FLINK-3988>) has been added for
> this work,
>
> All the infomation of DistributedLog, could be found at here:
> http://distributedlog.io.
>
> Thanks a lot.
> -Jia
>

[Tzu-Li Tai]

Hi Jia,

There's currently no dedicated documentation for composing streaming connectors, but for examples, I would say that the Elasticsearch and Nifi connectors are a good place to start learning. For a more full blown example I recommend taking a look at the Kafka connector. Basically, you'll extend the RichParallelSourceFunction for sources (or RichSinkFunction for sinks) and implement the required methods that will be called during the source / sink tasks' life cycle.

I'd be happy to help you with any questions you may come into while implementing the connector. Also, I'm thinking about adding documentation for composing a new connector (filed in https://issues.apache.org/jira/browse/FLINK-3989) as its a frequently asked question, so any feedback during your learning process will be quite helpful!

[Sijie Guo]

Ah, that's cool. Look forward to your flink connector. we are willing to
offer any helps on DL.

- Sijie

On Sat, May 28, 2016 at 9:33 PM, Jia Zhai <[hidden email]> wrote:

> +Distributedlog groups to get pertential help and instructions.
> Thanks.
>
> On Sun, May 29, 2016 at 11:33 AM, Jia Zhai <[hidden email]> wrote:
>
>> Hi,
>> I would like to add a connector to DistributedLog, which recently
>> published by twitter. :)
>> Is there any special instructions or best practise to add a new
>> connector?  Any suggestion is very appreciated.
>>
>> The JIRA tickets(FLINK-3987
>> <https://issues.apache.org/jira/browse/FLINK-3987> and FLINK-3988
>> <https://issues.apache.org/jira/browse/FLINK-3988>) has been added for
>> this work,
>>
>> All the infomation of DistributedLog, could be found at here:
>> http://distributedlog.io.
>>
>> Thanks a lot.
>> -Jia
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "distributedlog-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [hidden email].
> To post to this group, send email to [hidden email].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/distributedlog-user/CALsc%2BXqAUa__e3gUkCr%2BmVk7mASDeFFV-eBnFPgD47HhUrmC6g%40mail.gmail.com
> <https://groups.google.com/d/msgid/distributedlog-user/CALsc%2BXqAUa__e3gUkCr%2BmVk7mASDeFFV-eBnFPgD47HhUrmC6g%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

[Yijie Shen]

Hi Jia, I'm currently working on a stream connector for DL, do you mind if
we cooperate on FLINK-3987?
If you don't mind, I would do some clean up locally in the next days and
share it here.

Hi Sijie, I think we should publish DL jars to public repository in order
to be referenced by other project.
I've field https://github.com/twitter/distributedlog/issues/20 a couple
days ago.

Best,
Yijie

On Sun, May 29, 2016 at 2:55 PM, Sijie Guo <[hidden email]> wrote:

> Ah, that's cool. Look forward to your flink connector. we are willing to
> offer any helps on DL.
>
> - Sijie
>
> On Sat, May 28, 2016 at 9:33 PM, Jia Zhai <[hidden email]> wrote:
>
> > +Distributedlog groups to get pertential help and instructions.
> > Thanks.
> >
> > On Sun, May 29, 2016 at 11:33 AM, Jia Zhai <[hidden email]> wrote:
> >
> >> Hi,
> >> I would like to add a connector to DistributedLog, which recently
> >> published by twitter. :)
> >> Is there any special instructions or best practise to add a new
> >> connector?  Any suggestion is very appreciated.
> >>
> >> The JIRA tickets(FLINK-3987
> >> <https://issues.apache.org/jira/browse/FLINK-3987> and FLINK-3988
> >> <https://issues.apache.org/jira/browse/FLINK-3988>) has been added for
> >> this work,
> >>
> >> All the infomation of DistributedLog, could be found at here:
> >> http://distributedlog.io.
> >>
> >> Thanks a lot.
> >> -Jia
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "distributedlog-user" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [hidden email].
> > To post to this group, send email to
> [hidden email].
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/distributedlog-user/CALsc%2BXqAUa__e3gUkCr%2BmVk7mASDeFFV-eBnFPgD47HhUrmC6g%40mail.gmail.com
> > <
> https://groups.google.com/d/msgid/distributedlog-user/CALsc%2BXqAUa__e3gUkCr%2BmVk7mASDeFFV-eBnFPgD47HhUrmC6g%40mail.gmail.com?utm_medium=email&utm_source=footer
> >
> > .
> > For more options, visit https://groups.google.com/d/optout.
> >
>

[Jia Zhai]

Hi Gordon.
Thanks a lot for the information and the Jira ticket.

On Sun, May 29, 2016 at 1:37 PM, Tzu-Li (Gordon) Tai <[hidden email]>
wrote:

> Hi Jia,
>
> There's currently no dedicated documentation for composing streaming
> connectors, but for examples, I would say that the Elasticsearch and Nifi
> connectors are a good place to start learning. For a more full blown
> example
> I recommend taking a look at the Kafka connector. Basically, you'll extend
> the RichParallelSourceFunction for sources (or RichSinkFunction for sinks)
> and implement the required methods that will be called during the source /
> sink tasks' life cycle.
>
> I'd be happy to help you with any questions you may come into while
> implementing the connector. Also, I'm thinking about adding documentation
> for composing a new connector (filed in
> https://issues.apache.org/jira/browse/FLINK-3989) as its a frequently
> asked
> question, so any feedback during your learning process will be quite
> helpful!
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Would-like-to-add-a-connector-to-DistributedLog-tp11869p11872.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>