Hi,
I'm working on
https://issues.apache.org/jira/browse/FLINK-2186As I understand, Flink cannot read wide-column files in tuple, but pojo
So far we must create that pojo manually, it's convenient when count of columns not so many
When it's over thousand - hardly seems possible
To solve this issue I see these ways:
- Create an inputFormat that reads each column with proper type serializer and keeps them in common storage like Obejct[].
And keeps meta-information about field types. Some chunks of code of an attempt
https://github.com/apache/flink/compare/master...tonycox:FLINK-2186- Use a complex combination of Tuples or/and Pojos
- Somehow use a code generation to create a pojo with huge field count
What do you think?
Best regards,
Anton