Fwd: Re: AVRO Union type support in Flink

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Re: AVRO Union type support in Flink

Timo Walther-2
We have similar checks in our KafkaAvroTableSource, but I could not find
such a check in AvroTypeInfo. The field should have a generic type, so
you can work with it. If you want to use it as key, you might have to
use a mapper before and convert it into a valid key type.

Timo



-------- Weitergeleitete Nachricht --------
Betreff: Re: AVRO Union type support in Flink
Datum: Wed, 19 Jul 2017 10:26:24 -0400
Von: Vishnu Viswanath <[hidden email]>
An: Timo Walther <[hidden email]>



Hi Timo,

Thanks for checking that. I did not try yet. My current application uses
Cascading and it has the limitation that Union cannot contain two
concrete types - link
<https://github.com/ScaleUnlimited/cascading.avro/blob/master/scheme/src/main/java/cascading/avro/AvroToCascading.java#L137>,
so was wondering if I can use Flink. Will give it a try.

Hi Martin,
The documentation is here
<https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/connectors.html#avro-support-in-flink>
I use it to create AVRO files from source files in S3 and write to Kafka.

Thanks,
Vishnu


On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <[hidden email]
<mailto:[hidden email]>> wrote:

    Hi Vishnu,

    I took a look into the code. Actually, we should support it.
    However, those types might be mapped to Java Objects that will be
    serialized with our generic Kryo serializer. Have you tested it?

    Regards,
    Timo


    Am 19.07.17 um 06:30 schrieb Martin Eden:

>     Hey Vishnu,
>
>     For those of us on the list that are not very familiar with Flink
>     and Avro can you give a pointed to the docs you are referring to
>     and how you intend to use it? Just so we gain understanding as well.
>
>     Thanks,
>     Martin
>
>     On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath
>     <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         Hi All,
>
>         Does Flink support AVRO union types - Documentation says it
>         supports nullable types: {"name": "type_double_test", "type":
>         ["null", "double"]}
>
>         But my schema has something like : {"name": "union_field",
>         "type": ["string", "double"]}
>
>         Thanks
>         Vishnu
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Re: AVRO Union type support in Flink

Vishnu Viswanath
Hi Timo,

I just modified AvroOutputFormatTest to test this and it works fine!. I
don't plan to use it to key by, but it is a good point. Thanks.

Regards,
Vishnu

On Wed, Jul 19, 2017 at 10:57 AM, Timo Walther <[hidden email]> wrote:

> We have similar checks in our KafkaAvroTableSource, but I could not find
> such a check in AvroTypeInfo. The field should have a generic type, so you
> can work with it. If you want to use it as key, you might have to use a
> mapper before and convert it into a valid key type.
>
> Timo
>
>
> -------- Weitergeleitete Nachricht --------
> Betreff: Re: AVRO Union type support in Flink
> Datum: Wed, 19 Jul 2017 10:26:24 -0400
> Von: Vishnu Viswanath <[hidden email]>
> <[hidden email]>
> An: Timo Walther <[hidden email]> <[hidden email]>
>
>
> Hi Timo,
>
> Thanks for checking that. I did not try yet. My current application uses
> Cascading and it has the limitation that Union cannot contain two concrete
> types - link
> <https://github.com/ScaleUnlimited/cascading.avro/blob/master/scheme/src/main/java/cascading/avro/AvroToCascading.java#L137>,
> so was wondering if I can use Flink. Will give it a try.
>
> Hi Martin,
> The documentation is here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/connectors.html#avro-support-in-flink>
> I use it to create AVRO files from source files in S3 and write to Kafka.
>
> Thanks,
> Vishnu
>
>
> On Wed, Jul 19, 2017 at 5:55 AM, Timo Walther <[hidden email]> wrote:
>
>> Hi Vishnu,
>>
>> I took a look into the code. Actually, we should support it. However,
>> those types might be mapped to Java Objects that will be serialized with
>> our generic Kryo serializer. Have you tested it?
>>
>> Regards,
>> Timo
>>
>>
>> Am 19.07.17 um 06:30 schrieb Martin Eden:
>>
>> Hey Vishnu,
>>
>> For those of us on the list that are not very familiar with Flink and
>> Avro can you give a pointed to the docs you are referring to and how you
>> intend to use it? Just so we gain understanding as well.
>>
>> Thanks,
>> Martin
>>
>> On Tue, Jul 18, 2017 at 9:12 PM, Vishnu Viswanath <
>> [hidden email]> wrote:
>>
>>> Hi All,
>>>
>>> Does Flink support AVRO union types - Documentation says it supports
>>> nullable types: {"name": "type_double_test", "type": ["null", "double"]}
>>>
>>> But my schema has something like : {"name": "union_field", "type":
>>> ["string", "double"]}
>>>
>>> Thanks
>>> Vishnu
>>>
>>>
>>>
>>
>>
>