Flink cluster deployment strategy

[sidhant gupta]

Hi all,

I'm kind of new to flink cluster deployment. I wanted to know which flink
cluster deployment and which job mode in aws is better in terms of ease of
deployment, maintenance, HA, cost, etc. As of now I am considering aws EMR
vs ECS (docker containers). We have a usecase of setting up a data
streaming api which reads records from a Kafka topic, process it and then
write to a another Kafka topic. Please let me know your thoughts on this.

Thanks
Sidhant Gupta

[Till Rohrmann]

Hi Sidhant,

I am not an expert on AWS services but I believe that EMR might be a bit
easier to start with since AWS EMR comes with Flink support out of the box
[1]. On ECS I believe that you would have to set up the containers
yourself. Another interesting deployment option could be to use Flink's
native Kubernetes integration [2] which would work on AWS EKS.

[1]
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-create-cluster.html
[2]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html

Cheers,
Till

On Tue, Aug 11, 2020 at 9:16 AM sidhant gupta <[hidden email]> wrote:

> Hi all,
>
> I'm kind of new to flink cluster deployment. I wanted to know which flink
> cluster deployment and which job mode in aws is better in terms of ease of
> deployment, maintenance, HA, cost, etc. As of now I am considering aws EMR
> vs ECS (docker containers). We have a usecase of setting up a data
> streaming api which reads records from a Kafka topic, process it and then
> write to a another Kafka topic. Please let me know your thoughts on this.
>
> Thanks
> Sidhant Gupta
>