SDK vs Connectors

[Prasanna kumar]

Hi Team,

Following is the pipeline
Kafka => Processing => SNS Topics .

Flink Does not provide a SNS connector out of the box.

a) I implemented the above by using AWS SDK and published the messages in
the Map operator itself.
The pipeline is working well. I see messages flowing to SNS topics.

b) Another approach is that I could write a custom sink function and still
publish to SNS using SDK in this stage.

Questions
1) What would be the primary difference between approach a) and b). Is
there any significant advantage of one over the other ?

2) Would at least once guarantee be confirmed if we follow the above
approach?

3) Would there be any significant disadvantages(rather what we need to be
careful ) of writing our custom sink functions ?

Thanks,
Prasanna.

[Yun Gao]

Hi Prasanna,

   1) Semantically both a) and b) would be Ok. If the Custom sink could be chained with the map operator (I assume the map operator is the "Processing" in the graph), there should be also no much difference physically, if they could not chain, then writting a custom sink would cause another pass of network transferring, but the custom sink would be run in a different thread, thus much more computation resources could be exploited.
   2) To achieve at-least-once, you need to implment the "CheckpointedFunction" interface, and ensures flushing all the data to the outside systems when snapshotting states. Since if the checkpointing succeed, the previous data will not be replayed after failover, thus these pieces of data need to be ensured written out before the checkpoint succeeds.
   3) From my side I don't think there are significant disadvantages of writing custom sink functions.

Best,
 Yun


------------------------------------------------------------------
Sender:Prasanna kumar<[hidden email]>
Date:2020/08/22 02:00:51
Recipient:user<[hidden email]>; <[hidden email]>
Theme:SDK vs Connectors

Hi Team,

Following is the pipeline
Kafka => Processing => SNS Topics .

Flink Does not provide a SNS connector out of the box.

a) I implemented the above by using AWS SDK and published the messages in the Map operator itself.  
The pipeline is working well. I see messages flowing to SNS topics.

b) Another approach is that I could write a custom sink function and still publish to SNS using SDK in this stage.

Questions
1) What would be the primary difference between approach a) and b). Is there any significant advantage of one over the other ?

2) Would at least once guarantee be confirmed if we follow the above approach?

3) Would there be any significant disadvantages(rather what we need to be careful ) of writing our custom sink functions ?

Thanks,
Prasanna.

[Prasanna kumar]

Thanks for the Reply Yun,

I see that when I publish the messages to SNS from map operator, in case of
any errors I find the checkpointing mechanism takes care of "no data loss".

One scenario I could not replicate is that, the method from SDK unable to
send messages to SNS but remains silent not throwing any
errors/exceptions.In this case we may not confirm "at least once guarantee"
of delivery of messages.

Prasanna.

On Sun 23 Aug, 2020, 07:51 Yun Gao, <[hidden email]> wrote:

> Hi Prasanna,
>
>    1) Semantically both a) and b) would be Ok. If the Custom sink could be
> chained with the map operator (I assume the map operator is the
> "Processing" in the graph), there should be also no much difference
> physically, if they could not chain, then writting a custom sink would
> cause another pass of network transferring, but the custom sink would be
> run in a different thread, thus much more computation resources could be
> exploited.
>    2) To achieve at-least-once, you need to implment the
> "CheckpointedFunction" interface, and ensures flushing all the data to the
> outside systems when snapshotting states. Since if the checkpointing
> succeed, the previous data will not be replayed after failover, thus these
> pieces of data need to be ensured written out before the checkpoint
> succeeds.
>    3) From my side I don't think there are significant disadvantages of
> writing custom sink functions.
>
> Best,
>  Yun
>
>
> ------------------------------------------------------------------
> Sender:Prasanna kumar<[hidden email]>
> Date:2020/08/22 02:00:51
> Recipient:user<[hidden email]>; <[hidden email]>
> Theme:SDK vs Connectors
>
> Hi Team,
>
> Following is the pipeline
> Kafka => Processing => SNS Topics .
>
> Flink Does not provide a SNS connector out of the box.
>
> a) I implemented the above by using AWS SDK and published the messages in
> the Map operator itself.
> The pipeline is working well. I see messages flowing to SNS topics.
>
> b) Another approach is that I could write a custom sink function and still
> publish to SNS using SDK in this stage.
>
> Questions
> 1) What would be the primary difference between approach a) and b). Is
> there any significant advantage of one over the other ?
>
> 2) Would at least once guarantee be confirmed if we follow the above
> approach?
>
> 3) Would there be any significant disadvantages(rather what we need to be
> careful ) of writing our custom sink functions ?
>
> Thanks,
> Prasanna.
>
>