Flink Connector for AWS SQS

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink Connector for AWS SQS

Ray Ruvinskiy
I’m interested in writing a connector for AWS SQS (https://aws.amazon.com/sqs/). I’ve looked, and I don’t believe one is available. Flink does have a Kinesis collector, and there is some overlap in terms of basic AWS configuration handling (I believe mostly limited to what’s in https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/AWSConfigConstants.java). I have a couple of questions:

1. Should a potential new SQS connector be submitted to the flink project directly or to bahir-flink? I’m not entirely clear on the protocol. How is it decided which connector goes where?
2. Should I try to make use of the already existing AWSConfigConstants that have been written for Kinesis, or should I effectively replicate them? I can see potential for factoring them out to a common AWS class if both connectors were to live in flink proper, but that wouldn’t be possible if SQS lived in bahir-flink.

Thanks,

Ray


Reply | Threaded
Open this post in threaded view
|

Re: Flink Connector for AWS SQS

Chesnay Schepler-3
Hello,

There's isn't really a protocol for that; I would suggest to open the PR
against the Flink repo.

For now feel free to reuse the AWSConfigConstants, we can always
duplicate them later on if necessary.

Regards,
Chesnay

On 17.07.2017 16:50, Ray Ruvinskiy wrote:

> I’m interested in writing a connector for AWS SQS (https://aws.amazon.com/sqs/). I’ve looked, and I don’t believe one is available. Flink does have a Kinesis collector, and there is some overlap in terms of basic AWS configuration handling (I believe mostly limited to what’s in https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/config/AWSConfigConstants.java). I have a couple of questions:
>
> 1. Should a potential new SQS connector be submitted to the flink project directly or to bahir-flink? I’m not entirely clear on the protocol. How is it decided which connector goes where?
> 2. Should I try to make use of the already existing AWSConfigConstants that have been written for Kinesis, or should I effectively replicate them? I can see potential for factoring them out to a common AWS class if both connectors were to live in flink proper, but that wouldn’t be possible if SQS lived in bahir-flink.
>
> Thanks,
>
> Ray
>
>