We have similar checks in our KafkaAvroTableSource, but I could not find
such a check in AvroTypeInfo. The field should have a generic type, so you can work with it. If you want to use it as key, you might have to use a mapper before and convert it into a valid key type. Timo -------- Weitergeleitete Nachricht -------- Betreff: Re: AVRO Union type support in Flink Datum: Wed, 19 Jul 2017 10:26:24 -0400 Von: Vishnu Viswanath <[hidden email]> An: Timo Walther <[hidden email]> Hi Timo, Thanks for checking that. I did not try yet. My current application uses Cascading and it has the limitation that Union cannot contain two concrete types - link <https://github.com/ScaleUnlimited/cascading.avro/blob/master/scheme/src/main/java/cascading/avro/AvroToCascading.java#L137>, so was wondering if I can use Flink. Will give it a try. Hi Martin, The documentation is here <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/connectors.html#avro-support-in-flink> I use it to create AVRO files from source files in S3 and write to Kafka. Thanks, Vishnu On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <[hidden email] <mailto:[hidden email]>> wrote: Hi Vishnu, I took a look into the code. Actually, we should support it. However, those types might be mapped to Java Objects that will be serialized with our generic Kryo serializer. Have you tested it? Regards, Timo Am 19.07.17 um 06:30 schrieb Martin Eden: > Hey Vishnu, > > For those of us on the list that are not very familiar with Flink > and Avro can you give a pointed to the docs you are referring to > and how you intend to use it? Just so we gain understanding as well. > > Thanks, > Martin > > On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath > <[hidden email] > <mailto:[hidden email]>> wrote: > > Hi All, > > Does Flink support AVRO union types - Documentation says it > supports nullable types: {"name": "type_double_test", "type": > ["null", "double"]} > > But my schema has something like : {"name": "union_field", > "type": ["string", "double"]} > > Thanks > Vishnu > > > |
Hi Timo,
I just modified AvroOutputFormatTest to test this and it works fine!. I don't plan to use it to key by, but it is a good point. Thanks. Regards, Vishnu On Wed, Jul 19, 2017 at 10:57 AM, Timo Walther <[hidden email]> wrote: > We have similar checks in our KafkaAvroTableSource, but I could not find > such a check in AvroTypeInfo. The field should have a generic type, so you > can work with it. If you want to use it as key, you might have to use a > mapper before and convert it into a valid key type. > > Timo > > > -------- Weitergeleitete Nachricht -------- > Betreff: Re: AVRO Union type support in Flink > Datum: Wed, 19 Jul 2017 10:26:24 -0400 > Von: Vishnu Viswanath <[hidden email]> > <[hidden email]> > An: Timo Walther <[hidden email]> <[hidden email]> > > > Hi Timo, > > Thanks for checking that. I did not try yet. My current application uses > Cascading and it has the limitation that Union cannot contain two concrete > types - link > <https://github.com/ScaleUnlimited/cascading.avro/blob/master/scheme/src/main/java/cascading/avro/AvroToCascading.java#L137>, > so was wondering if I can use Flink. Will give it a try. > > Hi Martin, > The documentation is here > <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/connectors.html#avro-support-in-flink> > I use it to create AVRO files from source files in S3 and write to Kafka. > > Thanks, > Vishnu > > > On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <[hidden email]> wrote: > >> Hi Vishnu, >> >> I took a look into the code. Actually, we should support it. However, >> those types might be mapped to Java Objects that will be serialized with >> our generic Kryo serializer. Have you tested it? >> >> Regards, >> Timo >> >> >> Am 19.07.17 um 06:30 schrieb Martin Eden: >> >> Hey Vishnu, >> >> For those of us on the list that are not very familiar with Flink and >> Avro can you give a pointed to the docs you are referring to and how you >> intend to use it? Just so we gain understanding as well. >> >> Thanks, >> Martin >> >> On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath < >> [hidden email]> wrote: >> >>> Hi All, >>> >>> Does Flink support AVRO union types - Documentation says it supports >>> nullable types: {"name": "type_double_test", "type": ["null", "double"]} >>> >>> But my schema has something like : {"name": "union_field", "type": >>> ["string", "double"]} >>> >>> Thanks >>> Vishnu >>> >>> >>> >> >> > |
Free forum by Nabble | Edit this page |