Assign-evenly kafkaTopicPartitions of multiple topics to flinkKafkaConsumer subtask

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Assign-evenly kafkaTopicPartitions of multiple topics to flinkKafkaConsumer subtask

徐小龙



Hi, all:


      When we consume multiplue topic in one flinkKafkaConsumer, each topic partition count is less than the parallelism of the consumer, there is a problem we encounted is that, currently the partitionAssigner can evenly assign partition of one topic to subtask, but the assignment process is indepent between topic, so finally , we get some subtasks is in charge of much more partitions while some subtask is total free! To slow this issuse, i have reported the issue by https://issues.apache.org/jira/browse/FLINK-22840.
   
      I worked on this issue by add one assign strategy wich can deal with this requirement , and want to contibue to flink guys!


      Is there someone can talk about this issuce with me ?


   













--

顺祺!
徐小龙
同济大学软件学院
上海市曹安公路4800号济事楼508  201804
电话/传真:13671633451
E-MAIL:[hidden email]