Tzu-Li (Gordon) Tai created FLINK-16396:
-------------------------------------------
Summary: Support binding Kafka egresses with YAML-ized modules
Key: FLINK-16396
URL:
https://issues.apache.org/jira/browse/FLINK-16396 Project: Flink
Issue Type: New Feature
Components: Stateful Functions
Affects Versions: statefun-1.1
Reporter: Tzu-Li (Gordon) Tai
Assignee: Tzu-Li (Gordon) Tai
For a full multi-language story in Stateful Functions, we're still lacking a YAML-ized way to bind egresses. We should start with a Kafka one, as that is the currently only supported ingress for multi-language applications.
We propose to initially add a more "manual" approach for this, which gives the user full freedom on partitioning and record serialization.
A more specialized egress with built-in routing / protobuf serialization can maybe be added in the future once we collect more knowledge on usage from users.
The plan is as follows:
* Add a {{KafkaProducerRecord}} protobuf message with schema:
{code}
message KafkaProducerRecord {
string topic
string key
bytes valueBytes
}
{code}
How the value was serialized would be completely user-space; the bytes are written as is to Kafka. Partitioning depends on the partitioner used by the Kafka client, which may be configured with Kafka properties via YAML.
* The Flink sink provider for this egress should expect only messages of {{KafkaProducerRecord}} (wrapped within a protobuf Any). It simply transforms the information in the received {{KafkaProducerRecord}} to Kafka's own {{ProducerRecord}}, and writes that to Kafka.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Locked
