[DISCUSS] FLIP-86: Improve Connector Properties
Locked
[DISCUSS] FLIP-86: Improve Connector Properties
 
|
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Hi Jark, Majority of the changes make sense to me. I think they will be
helpful for the users. I have a slight concern about the kafka's connector.properties property :) . I wonder if we should
have them under a single key: connector.properties: `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092` As far as I understand it, these are mostly the properties that
are forwarded straight to the underlying KafkaClient. Such
properties are mostly defined and documented by the kafka itself
rather than flink. They might also follow a different formatting
scheme than we have for our properties. Moreover how do we decide
which properties are put into the Properties object and which are
not? I would be happy to hear what others think about that part,
as I am not convinced myself about that part. Best, Dawid On 13/11/2019 13:22, Jark Wu wrote:
Hi everyone, Connector properties is a very basic component which is used to construct a connector table via YAML, descriptor API, or DDL. It is also the serialization representation when persisting into catalog. However, we have encountered some problems when using connector properties, especially in DDL. This FLIP is aiming to solve following two issues: - FLINK-14645: Data types defined in DDL loses precision and nullability when converting to properties. - FLINK-14649: Some properties structure is hard to define in DDL, e.g. List and Map structure. This FLIP proposes new properties to represent data types in schema and to flatten properties in existing connectors. FLIP-86: https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit# Thanks for any feedback! Best, Jark |
signature.asc (849 bytes)
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Hi Jark, I see the not readable point and that is also my concern about my suggestion with all the properties under a single key as the list might get quite long. What do you think about modifying your suggestion a bit and at
least add the `properties` prefix? So that we can distinguish the
Kafka defined properties and the Flink's ones? What do you think about properties like: connector.properties.zookeeper.connect: .... connector.properties.bootstrap-servers: ....I think that's also what Konstantin was suggesting. I might be wrong on that though :) Best, Dawid
On 14/11/2019 04:52, Jark Wu wrote:
Hi Dawid, Konstantin, This is an interesting topic. I will list the prons and cons of these 2 options. 1) under a single key: `connector.properties` Cons: - Users can pass in arbitrary properties defined in kafka docs Prons: - The property value is collapsed together and will be long and not readable. - Some values may need to be escaped because of the separator. 2) flattened keys: `connector.zookeeper.connect`, `connector.bootstrap. servers` Cons: - Better validation and helpful exception message if the value is wrong. - Better documentation about which properties are required and what it means. - Easy to write. Prons: - If users want to pass in other kafka properties, need to support them and wait until the next major release. From my point of view, we can support both #1 and #2, but #2 will override the values defined in #1. So that, users can define the important properties in falttented keys, and can define unsupported properties in `connector.properties`. What do you think? Best, Jark On Thu, 14 Nov 2019 at 01:57, Konstantin Knauf [hidden email] wrote:Hi Dawid, Hi Jark, in my experience it is very important to be able to forward arbitrary properties to the underlying KafkaClient. This seems to be possible in both cases. I am leaning towards Jark's original suggestion. Flink's documentation would only need to state that it forwards everything under "connector.properties" to the KafkaClient. If the external system has a different formatting scheme, we could still document this on a per-connector basis (use Flink's formatting scheme and document how it is translated to the external system's native properties) or if possible use the external system's scheme under "connector.properties". I don't really know the limitations on our side w.r.t this. Cheers, Konstantin On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz [hidden email] wrote:Hi Jark, Majority of the changes make sense to me. I think they will be helpfulforthe users. I have a slight concern about the kafka's connector.properties property :) . I wonder if we should have them under a single key: connector.properties:`zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092`As far as I understand it, these are mostly the properties that are forwarded straight to the underlying KafkaClient. Such properties are mostly defined and documented by the kafka itself rather than flink. They might also follow a different formatting scheme than we have for our properties. Moreover how do we decide which properties are put into the Properties object and which are not? I would be happy to hear what others think about that part, as I am not convinced myself about that part. Best, Dawid On 13/11/2019 13:22, Jark Wu wrote: Hi everyone, Connector properties is a very basic component which is used toconstruct aconnector table via YAML, descriptor API, or DDL. It is also the serialization representation when persisting into catalog. However, wehaveencountered some problems when using connector properties, especially in DDL. This FLIP is aiming to solve following two issues: - FLINK-14645: Data types defined in DDL loses precision and nullability when converting to properties. - FLINK-14649: Some properties structure is hard to define in DDL, e.g. List and Map structure. This FLIP proposes new properties to represent data types in schema andtoflatten properties in existing connectors. FLIP-86:https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit#Thanks for any feedback! Best, Jark-- Konstantin Knauf | Solutions Architect +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Tony) Cheng |
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Yes, personally I would also go with the external system's format for
the exact reason you mentioned. Best, Dawid On 14/11/2019 10:32, Jark Wu wrote: > Hi Dawid, > > The `connector.properties.zookeeper.connect` sounds good to me. > > Just to clarify, in this way, we can support arbitrary properties and we > can document the important ones. > And regarding to the value format, shall we use the external system's > format or use Flink's format, e.g. the list separator? > I would prefer the external system's format, so that keys and values are > all forwarded directly. > > Best, > Jark > > > On Thu, 14 Nov 2019 at 17:02, Dawid Wysakowicz <[hidden email]> > wrote: > >> Hi Jark, >> >> I see the not readable point and that is also my concern about my >> suggestion with all the properties under a single key as the list might get >> quite long. >> >> What do you think about modifying your suggestion a bit and at least add >> the `properties` prefix? So that we can distinguish the Kafka defined >> properties and the Flink's ones? >> >> What do you think about properties like: >> >> connector.properties.zookeeper.connect: .... >> >> connector.properties.bootstrap-servers: .... >> >> >> I think that's also what Konstantin was suggesting. I might be wrong on >> that though :) >> >> Best, >> >> Dawid >> >> >> On 14/11/2019 04:52, Jark Wu wrote: >> >> Hi Dawid, Konstantin, >> >> This is an interesting topic. I will list the prons and cons of these 2 >> options. >> >> 1) under a single key: `connector.properties` >> Cons: >> - Users can pass in arbitrary properties defined in kafka docs >> Prons: >> - The property value is collapsed together and will be long and not >> readable. >> - Some values may need to be escaped because of the separator. >> >> 2) flattened keys: `connector.zookeeper.connect`, `connector.bootstrap. >> servers` >> Cons: >> - Better validation and helpful exception message if the value is wrong. >> - Better documentation about which properties are required and what it >> means. >> - Easy to write. >> Prons: >> - If users want to pass in other kafka properties, need to support them and >> wait until the next major release. >> >> From my point of view, we can support both #1 and #2, but #2 will override >> the values defined in #1. >> So that, users can define the important properties in falttented keys, and >> can define unsupported properties in `connector.properties`. >> >> What do you think? >> >> Best, >> Jark >> >> >> >> On Thu, 14 Nov 2019 at 01:57, Konstantin Knauf <[hidden email]> <[hidden email]> >> wrote: >> >> >> Hi Dawid, Hi Jark, >> >> in my experience it is very important to be able to forward arbitrary >> properties to the underlying KafkaClient. This seems to be possible in both >> cases. I am leaning towards Jark's original suggestion. Flink's >> documentation would only need to state that it forwards everything under >> "connector.properties" to the KafkaClient. >> >> If the external system has a different formatting scheme, we could still >> document this on a per-connector basis (use Flink's formatting scheme and >> document how it is translated to the external system's native properties) >> or if possible use the external system's scheme under >> "connector.properties". I don't really know the limitations on our side >> w.r.t this. >> >> Cheers, >> >> Konstantin >> >> >> >> On Wed, Nov 13, 2019 at 3:36 PM Dawid Wysakowicz <[hidden email]> <[hidden email]> >> wrote: >> >> >> Hi Jark, >> >> Majority of the changes make sense to me. I think they will be helpful >> >> for >> >> the users. I have a slight concern about the >> >> kafka's connector.properties property :) . I wonder if we should have >> them under a single key: >> >> connector.properties: >> >> `zookeeper.connect`:`localhost:2181`;`bootstrap.servers`:`localhost:9092` >> >> As far as I understand it, these are mostly the properties that are >> forwarded straight to the underlying KafkaClient. Such properties are >> mostly defined and documented by the kafka itself rather than flink. They >> might also follow a different formatting scheme than we have for our >> properties. Moreover how do we decide which properties are put into the >> Properties object and which are not? I would be happy to hear what others >> think about that part, as I am not convinced myself about that part. >> >> Best, >> >> Dawid >> On 13/11/2019 13:22, Jark Wu wrote: >> >> Hi everyone, >> >> Connector properties is a very basic component which is used to >> >> construct a >> >> connector table via YAML, descriptor API, or DDL. It is also the >> serialization representation when persisting into catalog. However, we >> >> have >> >> encountered some problems when using connector properties, especially in >> DDL. This FLIP is aiming to solve following two issues: >> >> - FLINK-14645: Data types defined in DDL loses precision and nullability >> when converting to properties. >> - FLINK-14649: Some properties structure is hard to define in DDL, e.g. >> List and Map structure. >> >> This FLIP proposes new properties to represent data types in schema and >> >> to >> >> flatten properties in existing connectors. >> >> FLIP-86: >> >> https://docs.google.com/document/d/14MlXFB45NUmUtesFMFZFjFhd4pqmYamxbfNwUPrfdL4/edit# >> >> Thanks for any feedback! >> >> Best, >> Jark >> >> >> >> >> -- >> >> Konstantin Knauf | Solutions Architect >> >> +49 160 91394525 >> >> >> Follow us @VervericaData Ververica <https://www.ververica.com/> <https://www.ververica.com/> >> >> >> -- >> >> Join Flink Forward <https://flink-forward.org/> <https://flink-forward.org/> - The Apache Flink >> Conference >> >> Stream Processing | Event Driven | Real Time >> >> -- >> >> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany >> >> -- >> Ververica GmbH >> Registered at Amtsgericht Charlottenburg: HRB 158244 B >> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji >> (Tony) Cheng >> >> >> |
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
Re: [DISCUSS] FLIP-86: Improve Connector Properties
|
|
| Free forum by Nabble | Edit this page |