[Dev] Dependencies issue related to implementing InputFormat Interface

[Pawan Manishka Gunarathna]

Hi,

I'm currently working on Flink InputFormat Interface implementation. I'm
writing a java program to read data from a file using InputputFormat
Interface. I used maven project and I have added following dependencies to
the pom.xml.

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-core</artifactId>
        <version>1.1.4</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_2.11</artifactId>
        <version>1.1.4</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>1.1.4</version>
    </dependency>

</dependencies>


I have a java class that implements InputFormat. It works with *InputFormat.
*But it didn't allow to used *InputFormat<OT, T extends InputSplit>. *That
OT field didn't recognized.

I need a any kind of help to solve this problem.

Thanks,
Pawan

--

*Pawan Gunaratne*
*Mob: [phone]*

[Chesnay Schepler-3]

Hello,

Did you write something like this?

    public class MyInputFormat implements InputFormat<OT, T extends
    InputSplit> {
         ....
    }

Regards,
Chesnay

On 17.01.2017 04:18, Pawan Manishka Gunarathna wrote:

> Hi,
>
> I'm currently working on Flink InputFormat Interface implementation. I'm
> writing a java program to read data from a file using InputputFormat
> Interface. I used maven project and I have added following dependencies to
> the pom.xml.
>
> <dependencies>
>      <dependency>
>          <groupId>org.apache.flink</groupId>
>          <artifactId>flink-core</artifactId>
>          <version>1.1.4</version>
>      </dependency>
>
>      <dependency>
>          <groupId>org.apache.flink</groupId>
>          <artifactId>flink-clients_2.11</artifactId>
>          <version>1.1.4</version>
>      </dependency>
>
>      <dependency>
>          <groupId>org.apache.flink</groupId>
>          <artifactId>flink-java</artifactId>
>          <version>1.1.4</version>
>      </dependency>
>
> </dependencies>
>
>
> I have a java class that implements InputFormat. It works with *InputFormat.
> *But it didn't allow to used *InputFormat<OT, T extends InputSplit>. *That
> OT field didn't recognized.
>
> I need a any kind of help to solve this problem.
>
> Thanks,
> Pawan
>

[Pawan Manishka Gunarathna]

Hi,
Yeah I also wrote in the way you have written......

public class ReadFromFile implements InputFormat<OT,T extends InputSplit>{
}

Is that a problem with that declaration or dependencies ?

Thanks,
Pawan

On Tue, Jan 17, 2017 at 7:56 PM, Chesnay Schepler <[hidden email]>
wrote:

> Hello,
>
> Did you write something like this?
>
>    public class MyInputFormat implements InputFormat<OT, T extends
>    InputSplit> {
>         ....
>    }
>
> Regards,
> Chesnay
>
> On 17.01.2017 04:18, Pawan Manishka Gunarathna wrote:
>
>> Hi,
>>
>> I'm currently working on Flink InputFormat Interface implementation. I'm
>> writing a java program to read data from a file using InputputFormat
>> Interface. I used maven project and I have added following dependencies to
>> the pom.xml.
>>
>> <dependencies>
>>      <dependency>
>>          <groupId>org.apache.flink</groupId>
>>          <artifactId>flink-core</artifactId>
>>          <version>1.1.4</version>
>>      </dependency>
>>
>>      <dependency>
>>          <groupId>org.apache.flink</groupId>
>>          <artifactId>flink-clients_2.11</artifactId>
>>          <version>1.1.4</version>
>>      </dependency>
>>
>>      <dependency>
>>          <groupId>org.apache.flink</groupId>
>>          <artifactId>flink-java</artifactId>
>>          <version>1.1.4</version>
>>      </dependency>
>>
>> </dependencies>
>>
>>
>> I have a java class that implements InputFormat. It works with
>> *InputFormat.
>> *But it didn't allow to used *InputFormat<OT, T extends InputSplit>. *That
>> OT field didn't recognized.
>>
>> I need a any kind of help to solve this problem.
>>
>> Thanks,
>> Pawan
>>
>>
>

--

*Pawan Gunaratne*
*Mob: [phone]*

[Fabian Hueske-2]

Hi Pawan,

If you want to read a file, you might want to extend the FileInputFormat
class. It has already a lot of file-related functionality implemented.
OT is the type of the records produced by the InputFormat. For example
Tuple2<String, Integer> if the input format produce a tuple with two fields
of String and Integer types.

Best, Fabian

2017-01-18 4:52 GMT+01:00 Pawan Manishka Gunarathna <
[hidden email]>:

> Hi,
> Yeah I also wrote in the way you have written......
>
> public class ReadFromFile implements InputFormat<OT,T extends InputSplit>{
> }
>
> Is that a problem with that declaration or dependencies ?
>
> Thanks,
> Pawan
>
> On Tue, Jan 17, 2017 at 7:56 PM, Chesnay Schepler <[hidden email]>
> wrote:
>
> > Hello,
> >
> > Did you write something like this?
> >
> >    public class MyInputFormat implements InputFormat<OT, T extends
> >    InputSplit> {
> >         ....
> >    }
> >
> > Regards,
> > Chesnay
> >
> > On 17.01.2017 04:18, Pawan Manishka Gunarathna wrote:
> >
> >> Hi,
> >>
> >> I'm currently working on Flink InputFormat Interface implementation. I'm
> >> writing a java program to read data from a file using InputputFormat
> >> Interface. I used maven project and I have added following dependencies
> to
> >> the pom.xml.
> >>
> >> <dependencies>
> >>      <dependency>
> >>          <groupId>org.apache.flink</groupId>
> >>          <artifactId>flink-core</artifactId>
> >>          <version>1.1.4</version>
> >>      </dependency>
> >>
> >>      <dependency>
> >>          <groupId>org.apache.flink</groupId>
> >>          <artifactId>flink-clients_2.11</artifactId>
> >>          <version>1.1.4</version>
> >>      </dependency>
> >>
> >>      <dependency>
> >>          <groupId>org.apache.flink</groupId>
> >>          <artifactId>flink-java</artifactId>
> >>          <version>1.1.4</version>
> >>      </dependency>
> >>
> >> </dependencies>
> >>
> >>
> >> I have a java class that implements InputFormat. It works with
> >> *InputFormat.
> >> *But it didn't allow to used *InputFormat<OT, T extends InputSplit>.
> *That
> >> OT field didn't recognized.
> >>
> >> I need a any kind of help to solve this problem.
> >>
> >> Thanks,
> >> Pawan
> >>
> >>
> >
>
>
> --
>
> *Pawan Gunaratne*
> *Mob: [phone]*
>

[Chesnay Schepler-3]

Hello,

The dependencies are fine.

The short answer is i would recommend you to read op on java generics.

The long answer is that OT and T are just placeholders for types that
are supposed to be replaced.

You can either provide the type in your implementation:
(in this example, the ReadFromFile inputformat returns the data as strings)

    public class ReadFromFile implements InputFormat<String, InputSplit>{
    }

or add a generic parameter to your inputformat:

    public class ReadFromFile<OT, T extends InputSplit > implements
    InputFormat<OT, T>{
    }

    In this case, the type would be specified when instantiating the
    Inputformat:

        new ReadFromFile<String, InputSplit>(...);

Note that you can also mix this; i.e define the InputSplit type (T) in
your implemenetation, but leave
OT to the user.

Regards,
Chesnay


On 18.01.2017 04:52, Pawan Manishka Gunarathna wrote:

> Hi,
> Yeah I also wrote in the way you have written......
>
> public class ReadFromFile implements InputFormat<OT,T extends InputSplit>{
> }
>
> Is that a problem with that declaration or dependencies ?
>
> Thanks,
> Pawan
>
> On Tue, Jan 17, 2017 at 7:56 PM, Chesnay Schepler <[hidden email]>
> wrote:
>
>> Hello,
>>
>> Did you write something like this?
>>
>>     public class MyInputFormat implements InputFormat<OT, T extends
>>     InputSplit> {
>>          ....
>>     }
>>
>> Regards,
>> Chesnay
>>
>> On 17.01.2017 04:18, Pawan Manishka Gunarathna wrote:
>>
>>> Hi,
>>>
>>> I'm currently working on Flink InputFormat Interface implementation. I'm
>>> writing a java program to read data from a file using InputputFormat
>>> Interface. I used maven project and I have added following dependencies to
>>> the pom.xml.
>>>
>>> <dependencies>
>>>       <dependency>
>>>           <groupId>org.apache.flink</groupId>
>>>           <artifactId>flink-core</artifactId>
>>>           <version>1.1.4</version>
>>>       </dependency>
>>>
>>>       <dependency>
>>>           <groupId>org.apache.flink</groupId>
>>>           <artifactId>flink-clients_2.11</artifactId>
>>>           <version>1.1.4</version>
>>>       </dependency>
>>>
>>>       <dependency>
>>>           <groupId>org.apache.flink</groupId>
>>>           <artifactId>flink-java</artifactId>
>>>           <version>1.1.4</version>
>>>       </dependency>
>>>
>>> </dependencies>
>>>
>>>
>>> I have a java class that implements InputFormat. It works with
>>> *InputFormat.
>>> *But it didn't allow to used *InputFormat<OT, T extends InputSplit>. *That
>>> OT field didn't recognized.
>>>
>>> I need a any kind of help to solve this problem.
>>>
>>> Thanks,
>>> Pawan
>>>
>>>
>

[Pawan Manishka Gunarathna]

Hi,

Thanks Fabian and Chesnay for providing those information.

Pawan

On Wed, Jan 18, 2017 at 2:11 PM, Chesnay Schepler <[hidden email]>
wrote:

> Hello,
>
> The dependencies are fine.
>
> The short answer is i would recommend you to read op on java generics.
>
> The long answer is that OT and T are just placeholders for types that are
> supposed to be replaced.
>
> You can either provide the type in your implementation:
> (in this example, the ReadFromFile inputformat returns the data as strings)
>
>    public class ReadFromFile implements InputFormat<String, InputSplit>{
>    }
>
> or add a generic parameter to your inputformat:
>
>    public class ReadFromFile<OT, T extends InputSplit > implements
>    InputFormat<OT, T>{
>    }
>
>    In this case, the type would be specified when instantiating the
>    Inputformat:
>
>        new ReadFromFile<String, InputSplit>(...);
>
> Note that you can also mix this; i.e define the InputSplit type (T) in
> your implemenetation, but leave
> OT to the user.
>
> Regards,
> Chesnay
>
>
>
> On 18.01.2017 04:52, Pawan Manishka Gunarathna wrote:
>
>> Hi,
>> Yeah I also wrote in the way you have written......
>>
>> public class ReadFromFile implements InputFormat<OT,T extends InputSplit>{
>> }
>>
>> Is that a problem with that declaration or dependencies ?
>>
>> Thanks,
>> Pawan
>>
>> On Tue, Jan 17, 2017 at 7:56 PM, Chesnay Schepler <[hidden email]>
>> wrote:
>>
>> Hello,
>>>
>>> Did you write something like this?
>>>
>>>     public class MyInputFormat implements InputFormat<OT, T extends
>>>     InputSplit> {
>>>          ....
>>>     }
>>>
>>> Regards,
>>> Chesnay
>>>
>>> On 17.01.2017 04:18, Pawan Manishka Gunarathna wrote:
>>>
>>> Hi,
>>>>
>>>> I'm currently working on Flink InputFormat Interface implementation. I'm
>>>> writing a java program to read data from a file using InputputFormat
>>>> Interface. I used maven project and I have added following dependencies
>>>> to
>>>> the pom.xml.
>>>>
>>>> <dependencies>
>>>>       <dependency>
>>>>           <groupId>org.apache.flink</groupId>
>>>>           <artifactId>flink-core</artifactId>
>>>>           <version>1.1.4</version>
>>>>       </dependency>
>>>>
>>>>       <dependency>
>>>>           <groupId>org.apache.flink</groupId>
>>>>           <artifactId>flink-clients_2.11</artifactId>
>>>>           <version>1.1.4</version>
>>>>       </dependency>
>>>>
>>>>       <dependency>
>>>>           <groupId>org.apache.flink</groupId>
>>>>           <artifactId>flink-java</artifactId>
>>>>           <version>1.1.4</version>
>>>>       </dependency>
>>>>
>>>> </dependencies>
>>>>
>>>>
>>>> I have a java class that implements InputFormat. It works with
>>>> *InputFormat.
>>>> *But it didn't allow to used *InputFormat<OT, T extends InputSplit>.
>>>> *That
>>>> OT field didn't recognized.
>>>>
>>>> I need a any kind of help to solve this problem.
>>>>
>>>> Thanks,
>>>> Pawan
>>>>
>>>>
>>>>
>>
>

--

*Pawan Gunaratne*
*Mob: [phone]*