Assign-evenly kafkaTopicPartitions of multiple topics to flinkKafkaConsumer subtask

Posted by 徐小龙 on Jun 17, 2021; 1:41pm
URL: http://deprecated-apache-flink-mailing-list-archive.368.s1.nabble.com/Assign-evenly-kafkaTopicPartitions-of-multiple-topics-to-flinkKafkaConsumer-subtask-tp51480.html




Hi, all:


      When we consume multiplue topic in one flinkKafkaConsumer, each topic partition count is less than the parallelism of the consumer, there is a problem we encounted is that, currently the partitionAssigner can evenly assign partition of one topic to subtask, but the assignment process is indepent between topic, so finally , we get some subtasks is in charge of much more partitions while some subtask is total free! To slow this issuse, i have reported the issue by https://issues.apache.org/jira/browse/FLINK-22840.
   
      I worked on this issue by add one assign strategy wich can deal with this requirement , and want to contibue to flink guys!


      Is there someone can talk about this issuce with me ?


   













--

顺祺！
徐小龙
同济大学软件学院
上海市曹安公路4800号济事楼508  201804
电话/传真：[phone]
E-MAIL:[hidden email]