Type extraction with generic variables

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Type extraction with generic variables

Vasiliki Kalavri
Hi all,

one of the operations we want to implement for the flink graph api is
mapVertices, i.e. applying a mapper to the vertices of the graph.
The current implementation assumes that the vertex value type remains the
same:
https://github.com/project-flink/flink-graph/blob/master/src/main/java/flink/graphs/Graph.java
.
However, we would like to be able to change the vertex value type. When
trying this out, we got this error:

Type of TypeVariable 'NV' in 'class flink.graphs.Graph$ApplyMapperToVertex'
could not be determined. This is most likely a type erasure problem. The
type extraction currently supports types with generic variables only in
cases where all variables in the return type can be deduced from the input
type(s).

How could we solve this and support changing the vertex value type?
Thank you!

Cheers,
Vasia.
Reply | Threaded
Open this post in threaded view
|

Re: Type extraction with generic variables

Timo Walther
Hi Vasiliki,

your error is a very common problem we have together with types. The
problem is that Java does type erasure, which means that

return vertices.map(new ApplyMapperToVertex<K, VV>(mapper));

becomes

return vertices.map(new ApplyMapperToVertex(mapper));

Therefore we don't have the types. But since we have "implements
MapFunction<Tuple2<K, VV>, Tuple2<K, VV>>" the type extractor infers the
output through the input (which is always known).

The return type must be provided manually e.g. by implementing the
interface ResultTypeQueryable or by the a new API operator (see my last
mailing list mail).

Regards,
Timo


On 30.10.2014 11:59, Vasiliki Kalavri wrote:

> Hi all,
>
> one of the operations we want to implement for the flink graph api is
> mapVertices, i.e. applying a mapper to the vertices of the graph.
> The current implementation assumes that the vertex value type remains the
> same:
> https://github.com/project-flink/flink-graph/blob/master/src/main/java/flink/graphs/Graph.java
> .
> However, we would like to be able to change the vertex value type. When
> trying this out, we got this error:
>
> Type of TypeVariable 'NV' in 'class flink.graphs.Graph$ApplyMapperToVertex'
> could not be determined. This is most likely a type erasure problem. The
> type extraction currently supports types with generic variables only in
> cases where all variables in the return type can be deduced from the input
> type(s).
>
> How could we solve this and support changing the vertex value type?
> Thank you!
>
> Cheers,
> Vasia.
>

Reply | Threaded
Open this post in threaded view
|

Re: Type extraction with generic variables

Vasiliki Kalavri
Hi Timo,

thank you for the explanation!
I guess I will try implementing ResultTypeQueryable then :)

Cheers,
Vasia.

On 30 October 2014 13:01, Timo Walther <[hidden email]> wrote:

> Hi Vasiliki,
>
> your error is a very common problem we have together with types. The
> problem is that Java does type erasure, which means that
>
> return vertices.map(new ApplyMapperToVertex<K, VV>(mapper));
>
> becomes
>
> return vertices.map(new ApplyMapperToVertex(mapper));
>
> Therefore we don't have the types. But since we have "implements
> MapFunction<Tuple2<K, VV>, Tuple2<K, VV>>" the type extractor infers the
> output through the input (which is always known).
>
> The return type must be provided manually e.g. by implementing the
> interface ResultTypeQueryable or by the a new API operator (see my last
> mailing list mail).
>
> Regards,
> Timo
>
>
>
> On 30.10.2014 11:59, Vasiliki Kalavri wrote:
>
>> Hi all,
>>
>> one of the operations we want to implement for the flink graph api is
>> mapVertices, i.e. applying a mapper to the vertices of the graph.
>> The current implementation assumes that the vertex value type remains the
>> same:
>> https://github.com/project-flink/flink-graph/blob/master/
>> src/main/java/flink/graphs/Graph.java
>> .
>> However, we would like to be able to change the vertex value type. When
>> trying this out, we got this error:
>>
>> Type of TypeVariable 'NV' in 'class flink.graphs.Graph$
>> ApplyMapperToVertex'
>> could not be determined. This is most likely a type erasure problem. The
>> type extraction currently supports types with generic variables only in
>> cases where all variables in the return type can be deduced from the input
>> type(s).
>>
>> How could we solve this and support changing the vertex value type?
>> Thank you!
>>
>> Cheers,
>> Vasia.
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Type extraction with generic variables

Vasiliki Kalavri
Hello again,

I tried out your suggestion and made a variation of the mapVertices method,
which takes the class of the new value as an argument.
In order to extract the type from the class, I use
the TypeExtractor.getForClass() method.
Is this what you had in mind?
When testing, this seems to work fine for basic types and for a custom
custom class I tried, but I get an error when trying to change the value
type to a Tuple. Do I have to deal with Tuples separately or is there a
better way to extract the type?

Thanks!

Cheers,
V.

On 31 October 2014 01:16, Vasiliki Kalavri <[hidden email]>
wrote:

> Hi Timo,
>
> thank you for the explanation!
> I guess I will try implementing ResultTypeQueryable then :)
>
> Cheers,
> Vasia.
>
> On 30 October 2014 13:01, Timo Walther <[hidden email]> wrote:
>
>> Hi Vasiliki,
>>
>> your error is a very common problem we have together with types. The
>> problem is that Java does type erasure, which means that
>>
>> return vertices.map(new ApplyMapperToVertex<K, VV>(mapper));
>>
>> becomes
>>
>> return vertices.map(new ApplyMapperToVertex(mapper));
>>
>> Therefore we don't have the types. But since we have "implements
>> MapFunction<Tuple2<K, VV>, Tuple2<K, VV>>" the type extractor infers the
>> output through the input (which is always known).
>>
>> The return type must be provided manually e.g. by implementing the
>> interface ResultTypeQueryable or by the a new API operator (see my last
>> mailing list mail).
>>
>> Regards,
>> Timo
>>
>>
>>
>> On 30.10.2014 11:59, Vasiliki Kalavri wrote:
>>
>>> Hi all,
>>>
>>> one of the operations we want to implement for the flink graph api is
>>> mapVertices, i.e. applying a mapper to the vertices of the graph.
>>> The current implementation assumes that the vertex value type remains the
>>> same:
>>> https://github.com/project-flink/flink-graph/blob/master/
>>> src/main/java/flink/graphs/Graph.java
>>> .
>>> However, we would like to be able to change the vertex value type. When
>>> trying this out, we got this error:
>>>
>>> Type of TypeVariable 'NV' in 'class flink.graphs.Graph$
>>> ApplyMapperToVertex'
>>> could not be determined. This is most likely a type erasure problem. The
>>> type extraction currently supports types with generic variables only in
>>> cases where all variables in the return type can be deduced from the
>>> input
>>> type(s).
>>>
>>> How could we solve this and support changing the vertex value type?
>>> Thank you!
>>>
>>> Cheers,
>>> Vasia.
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Type extraction with generic variables

Timo Walther
Hey,

the TypeExtractor.getForClass() is only intended for primitive types, if
you want to extract all kinds of types you should use
TypeExtractor.createTypeInfo(Type t) (a class is also a type). However,
I think it would be better if your method takes TypeInformation as an
argument instead of a class, because otherwise the user has to extend
tuple in order to provide the tuple field types.

Regards,
Timo

On 08.11.2014 19:04, Vasiliki Kalavri wrote:

> Hello again,
>
> I tried out your suggestion and made a variation of the mapVertices method,
> which takes the class of the new value as an argument.
> In order to extract the type from the class, I use
> the TypeExtractor.getForClass() method.
> Is this what you had in mind?
> When testing, this seems to work fine for basic types and for a custom
> custom class I tried, but I get an error when trying to change the value
> type to a Tuple. Do I have to deal with Tuples separately or is there a
> better way to extract the type?
>
> Thanks!
>
> Cheers,
> V.
>
> On 31 October 2014 01:16, Vasiliki Kalavri <[hidden email]>
> wrote:
>
>> Hi Timo,
>>
>> thank you for the explanation!
>> I guess I will try implementing ResultTypeQueryable then :)
>>
>> Cheers,
>> Vasia.
>>
>> On 30 October 2014 13:01, Timo Walther <[hidden email]> wrote:
>>
>>> Hi Vasiliki,
>>>
>>> your error is a very common problem we have together with types. The
>>> problem is that Java does type erasure, which means that
>>>
>>> return vertices.map(new ApplyMapperToVertex<K, VV>(mapper));
>>>
>>> becomes
>>>
>>> return vertices.map(new ApplyMapperToVertex(mapper));
>>>
>>> Therefore we don't have the types. But since we have "implements
>>> MapFunction<Tuple2<K, VV>, Tuple2<K, VV>>" the type extractor infers the
>>> output through the input (which is always known).
>>>
>>> The return type must be provided manually e.g. by implementing the
>>> interface ResultTypeQueryable or by the a new API operator (see my last
>>> mailing list mail).
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>>
>>> On 30.10.2014 11:59, Vasiliki Kalavri wrote:
>>>
>>>> Hi all,
>>>>
>>>> one of the operations we want to implement for the flink graph api is
>>>> mapVertices, i.e. applying a mapper to the vertices of the graph.
>>>> The current implementation assumes that the vertex value type remains the
>>>> same:
>>>> https://github.com/project-flink/flink-graph/blob/master/
>>>> src/main/java/flink/graphs/Graph.java
>>>> .
>>>> However, we would like to be able to change the vertex value type. When
>>>> trying this out, we got this error:
>>>>
>>>> Type of TypeVariable 'NV' in 'class flink.graphs.Graph$
>>>> ApplyMapperToVertex'
>>>> could not be determined. This is most likely a type erasure problem. The
>>>> type extraction currently supports types with generic variables only in
>>>> cases where all variables in the return type can be deduced from the
>>>> input
>>>> type(s).
>>>>
>>>> How could we solve this and support changing the vertex value type?
>>>> Thank you!
>>>>
>>>> Cheers,
>>>> Vasia.
>>>>
>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Type extraction with generic variables

Stephan Ewen
Hi Vasia!

I think in your case, it would be even better to use "getMapReturnTypes(
MapFunction<IN, OUT> mapInterface, TypeInformation<IN> inType)"

That way you can pass the function that you wrap (the one that maps the
vertex values) and get its produced type.


But I think the example shows that we need much better tooling for types
hidden in nested functions...

Stephan


On Mon, Nov 10, 2014 at 11:00 AM, Timo Walther <[hidden email]> wrote:

> Hey,
>
> the TypeExtractor.getForClass() is only intended for primitive types, if
> you want to extract all kinds of types you should use
> TypeExtractor.createTypeInfo(Type t) (a class is also a type). However, I
> think it would be better if your method takes TypeInformation as an
> argument instead of a class, because otherwise the user has to extend tuple
> in order to provide the tuple field types.
>
> Regards,
> Timo
>
>
> On 08.11.2014 19:04, Vasiliki Kalavri wrote:
>
>> Hello again,
>>
>> I tried out your suggestion and made a variation of the mapVertices
>> method,
>> which takes the class of the new value as an argument.
>> In order to extract the type from the class, I use
>> the TypeExtractor.getForClass() method.
>> Is this what you had in mind?
>> When testing, this seems to work fine for basic types and for a custom
>> custom class I tried, but I get an error when trying to change the value
>> type to a Tuple. Do I have to deal with Tuples separately or is there a
>> better way to extract the type?
>>
>> Thanks!
>>
>> Cheers,
>> V.
>>
>> On 31 October 2014 01:16, Vasiliki Kalavri <[hidden email]>
>> wrote:
>>
>>  Hi Timo,
>>>
>>> thank you for the explanation!
>>> I guess I will try implementing ResultTypeQueryable then :)
>>>
>>> Cheers,
>>> Vasia.
>>>
>>> On 30 October 2014 13:01, Timo Walther <[hidden email]> wrote:
>>>
>>>  Hi Vasiliki,
>>>>
>>>> your error is a very common problem we have together with types. The
>>>> problem is that Java does type erasure, which means that
>>>>
>>>> return vertices.map(new ApplyMapperToVertex<K, VV>(mapper));
>>>>
>>>> becomes
>>>>
>>>> return vertices.map(new ApplyMapperToVertex(mapper));
>>>>
>>>> Therefore we don't have the types. But since we have "implements
>>>> MapFunction<Tuple2<K, VV>, Tuple2<K, VV>>" the type extractor infers the
>>>> output through the input (which is always known).
>>>>
>>>> The return type must be provided manually e.g. by implementing the
>>>> interface ResultTypeQueryable or by the a new API operator (see my last
>>>> mailing list mail).
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>>
>>>> On 30.10.2014 11:59, Vasiliki Kalavri wrote:
>>>>
>>>>  Hi all,
>>>>>
>>>>> one of the operations we want to implement for the flink graph api is
>>>>> mapVertices, i.e. applying a mapper to the vertices of the graph.
>>>>> The current implementation assumes that the vertex value type remains
>>>>> the
>>>>> same:
>>>>> https://github.com/project-flink/flink-graph/blob/master/
>>>>> src/main/java/flink/graphs/Graph.java
>>>>> .
>>>>> However, we would like to be able to change the vertex value type. When
>>>>> trying this out, we got this error:
>>>>>
>>>>> Type of TypeVariable 'NV' in 'class flink.graphs.Graph$
>>>>> ApplyMapperToVertex'
>>>>> could not be determined. This is most likely a type erasure problem.
>>>>> The
>>>>> type extraction currently supports types with generic variables only in
>>>>> cases where all variables in the return type can be deduced from the
>>>>> input
>>>>> type(s).
>>>>>
>>>>> How could we solve this and support changing the vertex value type?
>>>>> Thank you!
>>>>>
>>>>> Cheers,
>>>>> Vasia.
>>>>>
>>>>>
>>>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Type extraction with generic variables

Vasiliki Kalavri
Thanks Stephan!

Indeed, this worked fine :)

Cheers,
V.

On 10 November 2014 12:26, Stephan Ewen <[hidden email]> wrote:

> Hi Vasia!
>
> I think in your case, it would be even better to use "getMapReturnTypes(
> MapFunction<IN, OUT> mapInterface, TypeInformation<IN> inType)"
>
> That way you can pass the function that you wrap (the one that maps the
> vertex values) and get its produced type.
>
>
> But I think the example shows that we need much better tooling for types
> hidden in nested functions...
>
> Stephan
>
>
> On Mon, Nov 10, 2014 at 11:00 AM, Timo Walther <[hidden email]> wrote:
>
> > Hey,
> >
> > the TypeExtractor.getForClass() is only intended for primitive types, if
> > you want to extract all kinds of types you should use
> > TypeExtractor.createTypeInfo(Type t) (a class is also a type). However, I
> > think it would be better if your method takes TypeInformation as an
> > argument instead of a class, because otherwise the user has to extend
> tuple
> > in order to provide the tuple field types.
> >
> > Regards,
> > Timo
> >
> >
> > On 08.11.2014 19:04, Vasiliki Kalavri wrote:
> >
> >> Hello again,
> >>
> >> I tried out your suggestion and made a variation of the mapVertices
> >> method,
> >> which takes the class of the new value as an argument.
> >> In order to extract the type from the class, I use
> >> the TypeExtractor.getForClass() method.
> >> Is this what you had in mind?
> >> When testing, this seems to work fine for basic types and for a custom
> >> custom class I tried, but I get an error when trying to change the value
> >> type to a Tuple. Do I have to deal with Tuples separately or is there a
> >> better way to extract the type?
> >>
> >> Thanks!
> >>
> >> Cheers,
> >> V.
> >>
> >> On 31 October 2014 01:16, Vasiliki Kalavri <[hidden email]>
> >> wrote:
> >>
> >>  Hi Timo,
> >>>
> >>> thank you for the explanation!
> >>> I guess I will try implementing ResultTypeQueryable then :)
> >>>
> >>> Cheers,
> >>> Vasia.
> >>>
> >>> On 30 October 2014 13:01, Timo Walther <[hidden email]> wrote:
> >>>
> >>>  Hi Vasiliki,
> >>>>
> >>>> your error is a very common problem we have together with types. The
> >>>> problem is that Java does type erasure, which means that
> >>>>
> >>>> return vertices.map(new ApplyMapperToVertex<K, VV>(mapper));
> >>>>
> >>>> becomes
> >>>>
> >>>> return vertices.map(new ApplyMapperToVertex(mapper));
> >>>>
> >>>> Therefore we don't have the types. But since we have "implements
> >>>> MapFunction<Tuple2<K, VV>, Tuple2<K, VV>>" the type extractor infers
> the
> >>>> output through the input (which is always known).
> >>>>
> >>>> The return type must be provided manually e.g. by implementing the
> >>>> interface ResultTypeQueryable or by the a new API operator (see my
> last
> >>>> mailing list mail).
> >>>>
> >>>> Regards,
> >>>> Timo
> >>>>
> >>>>
> >>>>
> >>>> On 30.10.2014 11:59, Vasiliki Kalavri wrote:
> >>>>
> >>>>  Hi all,
> >>>>>
> >>>>> one of the operations we want to implement for the flink graph api is
> >>>>> mapVertices, i.e. applying a mapper to the vertices of the graph.
> >>>>> The current implementation assumes that the vertex value type remains
> >>>>> the
> >>>>> same:
> >>>>> https://github.com/project-flink/flink-graph/blob/master/
> >>>>> src/main/java/flink/graphs/Graph.java
> >>>>> .
> >>>>> However, we would like to be able to change the vertex value type.
> When
> >>>>> trying this out, we got this error:
> >>>>>
> >>>>> Type of TypeVariable 'NV' in 'class flink.graphs.Graph$
> >>>>> ApplyMapperToVertex'
> >>>>> could not be determined. This is most likely a type erasure problem.
> >>>>> The
> >>>>> type extraction currently supports types with generic variables only
> in
> >>>>> cases where all variables in the return type can be deduced from the
> >>>>> input
> >>>>> type(s).
> >>>>>
> >>>>> How could we solve this and support changing the vertex value type?
> >>>>> Thank you!
> >>>>>
> >>>>> Cheers,
> >>>>> Vasia.
> >>>>>
> >>>>>
> >>>>>
> >
>