James B. Fitzgerald created FLINK-14523:
-------------------------------------------
Summary: Flink Stops Consuming from Kafka after Leader Election
Key: FLINK-14523
URL:
https://issues.apache.org/jira/browse/FLINK-14523 Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Environment: In AWS we run the app on EMR with the following versions:
* EMR Release - emr-5.15.0
* Hadoop distribution - Amazon 2.8.3
* Flink - Flink 1.4.2
We submit the job to the cluster as an EMR step using ***command-runner.jar*. We submit the job with the following arguments:
{code:java}
"Args": [
"flink", "run", "-m", "yarn-cluster",
"-c", "com.salesforce.sde.streamingsearches.StreamingSearchesJob",
"-yst", "-ys", "4", "-yn", "10", "-yjm", "2800", "-ytm", "2800",
"-ynm", "streaming-searches-prod",
"-d", "/home/hadoop/streaming-searches-1.0-SNAPSHOT.jar"
]{code}
Additionally we build our application jar with Flink 1.4.2 and Kafka 0.11.
Reporter: James B. Fitzgerald
We have a Flink application running in AWS on EMR that streams input from a single Kafka topic. Whenever there is a Kafka leader election for any partition of the input topic, our Flink application stops consuming from Kafka entirely. To begin consuming from Kafka again the YARN app must be killed and restarted. We run this same application on premises and in AWS on EMR. We have only observed this behavior when it is running on EMR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
