[jira] [Created] (FLINK-4789) Avoid Kafka partition discovery on restore and share consumer instance for discovery and data consumption

[Shang Yuanchun (Jira)]

Robert Metzger created FLINK-4789:
-------------------------------------

             Summary: Avoid Kafka partition discovery on restore and share consumer instance for discovery and data consumption
                 Key: FLINK-4789
                 URL: https://issues.apache.org/jira/browse/FLINK-4789
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
    Affects Versions: 1.2.0
            Reporter: Robert Metzger


As part of FLINK-4379, the Kafka partition discovery was moved from the Constructor to the open() method. This is in general a good change, as outlined in FLINK-4155, as it allows us to detect new partitions and topics based on regex on the fly.

However, currently the partitions are discovered on restore as well.
Also, the {{FlinkKafkaConsumer09.getKafkaPartitions()}} is creating a separate {{KafkaConsumer}} just for the partition discovery.
Since the partition discovery happens on the task managers now, we can use the regular {{KafkaConsumer}} instance, which is used for data retrieval as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)