[DISCUSS] Python API for Fllink libraries

[Ivan Mushketyk]

Hi Flink developers,

It seems to me that Flink has two important "selling points":

1. It has Java, Scala and Python APIs
2. I has a number of useful libraries (ML, Gelly, CEP, and Table)

But as far as I understand, currently users cannot use any of these
libraries using a Python API. It seems to be a gap worth filling.

What do you think about it? Does it make sense to add CEP/Gelly/ML/Table
Python APIs?

Best regards,
Ivan.

[Chesnay Schepler-3]

Hello,

I would say no, as the general reception of the Python API was
*tumbleweed* so far.

In my opinion this would just lead to a massive increase in code to
maintain; we would need at least 2-3 active long-term python contributors.
Especially so since ML, CEP and Table are afaik still in heavy development.

If anything, before thinking about porting the libraries to python it
would make more sense to implement a python streaming API.
Or maybe /finish/ porting the DataSet API...

Regards,
Chesnay

On 19.08.2016 22:07, Ivan Mushketyk wrote:

> Hi Flink developers,
>
> It seems to me that Flink has two important "selling points":
>
> 1. It has Java, Scala and Python APIs
> 2. I has a number of useful libraries (ML, Gelly, CEP, and Table)
>
> But as far as I understand, currently users cannot use any of these
> libraries using a Python API. It seems to be a gap worth filling.
>
> What do you think about it? Does it make sense to add CEP/Gelly/ML/Table
> Python APIs?
>
> Best regards,
> Ivan.
>

[Ivan Mushketyk]

Hi Chesnay,

Thank you for you repply.
Out of curiosity, do you know why Python API reception was  *tumbleweed*?

Regarding the Python API, do you know what specifically should be done
there? I have some Python background so I was considering to contribute,
but I didn't find much tasks in the "Python" component:
https://issues.apache.org/jira/browse/FLINK-1926?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python%20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC

Best regards,
Ivan.


On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler <[hidden email]> wrote:

> Hello,
>
> I would say no, as the general reception of the Python API was
> *tumbleweed* so far.
>
> In my opinion this would just lead to a massive increase in code to
> maintain; we would need at least 2-3 active long-term python contributors.
> Especially so since ML, CEP and Table are afaik still in heavy development.
>
> If anything, before thinking about porting the libraries to python it
> would make more sense to implement a python streaming API.
> Or maybe /finish/ porting the DataSet API...
>
> Regards,
> Chesnay
>
> On 19.08.2016 22:07, Ivan Mushketyk wrote:
> > Hi Flink developers,
> >
> > It seems to me that Flink has two important "selling points":
> >
> > 1. It has Java, Scala and Python APIs
> > 2. I has a number of useful libraries (ML, Gelly, CEP, and Table)
> >
> > But as far as I understand, currently users cannot use any of these
> > libraries using a Python API. It seems to be a gap worth filling.
> >
> > What do you think about it? Does it make sense to add CEP/Gelly/ML/Table
> > Python APIs?
> >
> > Best regards,
> > Ivan.
> >
>
>

[Greg Hogan]

Hi Ivan,

My expectation would be that programs written for the Python API would be
much slower than when implementing with Java or Scala. A performance
comparison would be quite interesting. Gelly has both iterative and
non-iterative algorithms.

Greg

On Sat, Aug 20, 2016 at 7:11 PM, Ivan Mushketyk <[hidden email]>
wrote:

> Hi Chesnay,
>
> Thank you for you repply.
> Out of curiosity, do you know why Python API reception was  *tumbleweed*?
>
> Regarding the Python API, do you know what specifically should be done
> there? I have some Python background so I was considering to contribute,
> but I didn't find much tasks in the "Python" component:
> https://issues.apache.org/jira/browse/FLINK-1926?jql=
> project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python%
> 20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%
> 20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> Best regards,
> Ivan.
>
>
> On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler <[hidden email]> wrote:
>
> > Hello,
> >
> > I would say no, as the general reception of the Python API was
> > *tumbleweed* so far.
> >
> > In my opinion this would just lead to a massive increase in code to
> > maintain; we would need at least 2-3 active long-term python
> contributors.
> > Especially so since ML, CEP and Table are afaik still in heavy
> development.
> >
> > If anything, before thinking about porting the libraries to python it
> > would make more sense to implement a python streaming API.
> > Or maybe /finish/ porting the DataSet API...
> >
> > Regards,
> > Chesnay
> >
> > On 19.08.2016 22:07, Ivan Mushketyk wrote:
> > > Hi Flink developers,
> > >
> > > It seems to me that Flink has two important "selling points":
> > >
> > > 1. It has Java, Scala and Python APIs
> > > 2. I has a number of useful libraries (ML, Gelly, CEP, and Table)
> > >
> > > But as far as I understand, currently users cannot use any of these
> > > libraries using a Python API. It seems to be a gap worth filling.
> > >
> > > What do you think about it? Does it make sense to add
> CEP/Gelly/ML/Table
> > > Python APIs?
> > >
> > > Best regards,
> > > Ivan.
> > >
> >
> >
>

[Chesnay Schepler-3]

Hello Greg,

While I generally agree with you, there have been cases reported where
the Python API was actually faster due to the usage of C libraries.

Regards,
Chesnay

On 22.08.2016 16:21, Greg Hogan wrote:

> Hi Ivan,
>
> My expectation would be that programs written for the Python API would be
> much slower than when implementing with Java or Scala. A performance
> comparison would be quite interesting. Gelly has both iterative and
> non-iterative algorithms.
>
> Greg
>
> On Sat, Aug 20, 2016 at 7:11 PM, Ivan Mushketyk <[hidden email]>
> wrote:
>
>> Hi Chesnay,
>>
>> Thank you for you repply.
>> Out of curiosity, do you know why Python API reception was  *tumbleweed*?
>>
>> Regarding the Python API, do you know what specifically should be done
>> there? I have some Python background so I was considering to contribute,
>> but I didn't find much tasks in the "Python" component:
>> https://issues.apache.org/jira/browse/FLINK-1926?jql=
>> project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python%
>> 20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%
>> 20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>>
>> Best regards,
>> Ivan.
>>
>>
>> On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler <[hidden email]> wrote:
>>
>>> Hello,
>>>
>>> I would say no, as the general reception of the Python API was
>>> *tumbleweed* so far.
>>>
>>> In my opinion this would just lead to a massive increase in code to
>>> maintain; we would need at least 2-3 active long-term python
>> contributors.
>>> Especially so since ML, CEP and Table are afaik still in heavy
>> development.
>>> If anything, before thinking about porting the libraries to python it
>>> would make more sense to implement a python streaming API.
>>> Or maybe /finish/ porting the DataSet API...
>>>
>>> Regards,
>>> Chesnay
>>>
>>> On 19.08.2016 22:07, Ivan Mushketyk wrote:
>>>> Hi Flink developers,
>>>>
>>>> It seems to me that Flink has two important "selling points":
>>>>
>>>> 1. It has Java, Scala and Python APIs
>>>> 2. I has a number of useful libraries (ML, Gelly, CEP, and Table)
>>>>
>>>> But as far as I understand, currently users cannot use any of these
>>>> libraries using a Python API. It seems to be a gap worth filling.
>>>>
>>>> What do you think about it? Does it make sense to add
>> CEP/Gelly/ML/Table
>>>> Python APIs?
>>>>
>>>> Best regards,
>>>> Ivan.
>>>>
>>>

[Chesnay Schepler-3]

Helli Ivan,

i don't know why it is the way it is.

Regarding issues to work on: You should be able to go through the
transformations documentation and see which are not supported.

Regards,
Chesnay

On 21.08.2016 01:11, Ivan Mushketyk wrote:

> Hi Chesnay,
>
> Thank you for you repply.
> Out of curiosity, do you know why Python API reception was  *tumbleweed*?
>
> Regarding the Python API, do you know what specifically should be done
> there? I have some Python background so I was considering to contribute,
> but I didn't find much tasks in the "Python" component:
> https://issues.apache.org/jira/browse/FLINK-1926?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Python%20API%22%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> Best regards,
> Ivan.
>
>
> On Fri, 19 Aug 2016 at 22:45 Chesnay Schepler <[hidden email]> wrote:
>
>> Hello,
>>
>> I would say no, as the general reception of the Python API was
>> *tumbleweed* so far.
>>
>> In my opinion this would just lead to a massive increase in code to
>> maintain; we would need at least 2-3 active long-term python contributors.
>> Especially so since ML, CEP and Table are afaik still in heavy development.
>>
>> If anything, before thinking about porting the libraries to python it
>> would make more sense to implement a python streaming API.
>> Or maybe /finish/ porting the DataSet API...
>>
>> Regards,
>> Chesnay
>>
>> On 19.08.2016 22:07, Ivan Mushketyk wrote:
>>> Hi Flink developers,
>>>
>>> It seems to me that Flink has two important "selling points":
>>>
>>> 1. It has Java, Scala and Python APIs
>>> 2. I has a number of useful libraries (ML, Gelly, CEP, and Table)
>>>
>>> But as far as I understand, currently users cannot use any of these
>>> libraries using a Python API. It seems to be a gap worth filling.
>>>
>>> What do you think about it? Does it make sense to add CEP/Gelly/ML/Table
>>> Python APIs?
>>>
>>> Best regards,
>>> Ivan.
>>>
>>