Backwards compatibility for REST API

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Backwards compatibility for REST API

Thomas Weise
Hi,

Are there any compatibility guarantees for the Flink REST API or plans to
establish the same?

I'm asking since I noticed that the 1.5.1 release has an incompatible
change that prevents a 1.5.0 client to submit a job to 1.5.0 server
(FLINK-9280, removed endpoint /blobserver/port).

Such change is especially unexpected in a patch release, but even if it was
in 1.6.x, it is still problematic because it forces clients to upgrade.

Thanks,
Thomas
Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Chesnay Schepler-3
Effectively there are no guarantees for backwards compatibility,
although we try to maintain it.

There are plans to version the REST API
(https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
timeline.

The change you mentioned in particular primarily affects internal code
(since the removed endpoint is usually only used by the flink CLI).
Note we do not provide any guarantee that Flink components interact
properly if the versions don't exactly match, so you should always upgrade
the clients anyway.

On 24.07.2018 16:43, Thomas Weise wrote:

> Hi,
>
> Are there any compatibility guarantees for the Flink REST API or plans to
> establish the same?
>
> I'm asking since I noticed that the 1.5.1 release has an incompatible
> change that prevents a 1.5.0 client to submit a job to 1.5.0 server
> (FLINK-9280, removed endpoint /blobserver/port).
>
> Such change is especially unexpected in a patch release, but even if it was
> in 1.6.x, it is still problematic because it forces clients to upgrade.
>
> Thanks,
> Thomas
>

Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Thomas Weise
I would expect more external dependencies on the REST API and suggest to
consider versioning and compatibility guarantees.

The particular issue is from Beam and tracked here:
https://issues.apache.org/jira/browse/BEAM-4842

I would be surprised if that is the only instance besides Flink CLI trying
to use the REST API for job submission. In the case of Beam, it is
problematic to assume that the job service that contains the Flink client
will always be upgraded lock step with the server.

What other factors prevent interoperability between Flink versions? Will it
be possible to submit a job graph built by 1.5.1 client (that does not
contain any connectors or operators that depend on internal API) to 1.6.x
job manager?

Thanks,
Thomas


On Tue, Jul 24, 2018 at 7:58 AM Chesnay Schepler <[hidden email]> wrote:

> Effectively there are no guarantees for backwards compatibility,
> although we try to maintain it.
>
> There are plans to version the REST API
> (https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
> timeline.
>
> The change you mentioned in particular primarily affects internal code
> (since the removed endpoint is usually only used by the flink CLI).
> Note we do not provide any guarantee that Flink components interact
> properly if the versions don't exactly match, so you should always upgrade
> the clients anyway.
>
> On 24.07.2018 16:43, Thomas Weise wrote:
> > Hi,
> >
> > Are there any compatibility guarantees for the Flink REST API or plans to
> > establish the same?
> >
> > I'm asking since I noticed that the 1.5.1 release has an incompatible
> > change that prevents a 1.5.0 client to submit a job to 1.5.0 server
> > (FLINK-9280, removed endpoint /blobserver/port).
> >
> > Such change is especially unexpected in a patch release, but even if it
> was
> > in 1.6.x, it is still problematic because it forces clients to upgrade.
> >
> > Thanks,
> > Thomas
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Chesnay Schepler-3
/Will it be possible to submit a job graph built by 1.5.1 client (that
does not contain any connectors or operators that depend on internal
API) to 1.6.x job manager?/

Probably yes, but generally there's no guarantee.

The JobGraph is not a public API and thus may change in ways that could
cause serialization failures if the client and server versions do not match.

On 24.07.2018 17:12, Thomas Weise wrote:

> I would expect more external dependencies on the REST API and suggest to
> consider versioning and compatibility guarantees.
>
> The particular issue is from Beam and tracked here:
> https://issues.apache.org/jira/browse/BEAM-4842
>
> I would be surprised if that is the only instance besides Flink CLI trying
> to use the REST API for job submission. In the case of Beam, it is
> problematic to assume that the job service that contains the Flink client
> will always be upgraded lock step with the server.
>
> What other factors prevent interoperability between Flink versions? Will it
> be possible to submit a job graph built by 1.5.1 client (that does not
> contain any connectors or operators that depend on internal API) to 1.6.x
> job manager?
>
> Thanks,
> Thomas
>
>
> On Tue, Jul 24, 2018 at 7:58 AM Chesnay Schepler <[hidden email]> wrote:
>
>> Effectively there are no guarantees for backwards compatibility,
>> although we try to maintain it.
>>
>> There are plans to version the REST API
>> (https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
>> timeline.
>>
>> The change you mentioned in particular primarily affects internal code
>> (since the removed endpoint is usually only used by the flink CLI).
>> Note we do not provide any guarantee that Flink components interact
>> properly if the versions don't exactly match, so you should always upgrade
>> the clients anyway.
>>
>> On 24.07.2018 16:43, Thomas Weise wrote:
>>> Hi,
>>>
>>> Are there any compatibility guarantees for the Flink REST API or plans to
>>> establish the same?
>>>
>>> I'm asking since I noticed that the 1.5.1 release has an incompatible
>>> change that prevents a 1.5.0 client to submit a job to 1.5.0 server
>>> (FLINK-9280, removed endpoint /blobserver/port).
>>>
>>> Such change is especially unexpected in a patch release, but even if it
>> was
>>> in 1.6.x, it is still problematic because it forces clients to upgrade.
>>>
>>> Thanks,
>>> Thomas
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Chesnay Schepler-3
Note that the changes made to the REST API will in the long-term reduce
friction between versions,
as we no longer have to submit data through the blob service, which is
another internal API.

On 24.07.2018 17:25, Chesnay Schepler wrote:

> /Will it be possible to submit a job graph built by 1.5.1 client (that
> does not contain any connectors or operators that depend on internal
> API) to 1.6.x job manager?/
>
> Probably yes, but generally there's no guarantee.
>
> The JobGraph is not a public API and thus may change in ways that could
> cause serialization failures if the client and server versions do not
> match.
>
> On 24.07.2018 17:12, Thomas Weise wrote:
>> I would expect more external dependencies on the REST API and suggest to
>> consider versioning and compatibility guarantees.
>>
>> The particular issue is from Beam and tracked here:
>> https://issues.apache.org/jira/browse/BEAM-4842
>>
>> I would be surprised if that is the only instance besides Flink CLI
>> trying
>> to use the REST API for job submission. In the case of Beam, it is
>> problematic to assume that the job service that contains the Flink
>> client
>> will always be upgraded lock step with the server.
>>
>> What other factors prevent interoperability between Flink versions?
>> Will it
>> be possible to submit a job graph built by 1.5.1 client (that does not
>> contain any connectors or operators that depend on internal API) to
>> 1.6.x
>> job manager?
>>
>> Thanks,
>> Thomas
>>
>>
>> On Tue, Jul 24, 2018 at 7:58 AM Chesnay Schepler <[hidden email]>
>> wrote:
>>
>>> Effectively there are no guarantees for backwards compatibility,
>>> although we try to maintain it.
>>>
>>> There are plans to version the REST API
>>> (https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
>>> timeline.
>>>
>>> The change you mentioned in particular primarily affects internal code
>>> (since the removed endpoint is usually only used by the flink CLI).
>>> Note we do not provide any guarantee that Flink components interact
>>> properly if the versions don't exactly match, so you should always
>>> upgrade
>>> the clients anyway.
>>>
>>> On 24.07.2018 16:43, Thomas Weise wrote:
>>>> Hi,
>>>>
>>>> Are there any compatibility guarantees for the Flink REST API or
>>>> plans to
>>>> establish the same?
>>>>
>>>> I'm asking since I noticed that the 1.5.1 release has an incompatible
>>>> change that prevents a 1.5.0 client to submit a job to 1.5.0 server
>>>> (FLINK-9280, removed endpoint /blobserver/port).
>>>>
>>>> Such change is especially unexpected in a patch release, but even
>>>> if it
>>> was
>>>> in 1.6.x, it is still problematic because it forces clients to
>>>> upgrade.
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Chesnay Schepler-3
Looked some more into it, as of _right now_ a 1.5.X client will not work
against 1.6.0 cluster
since the submission body was extended. But we could add a guard against
this.

On 24.07.2018 17:27, Chesnay Schepler wrote:

> Note that the changes made to the REST API will in the long-term
> reduce friction between versions,
> as we no longer have to submit data through the blob service, which is
> another internal API.
>
> On 24.07.2018 17:25, Chesnay Schepler wrote:
>> /Will it be possible to submit a job graph built by 1.5.1 client
>> (that does not contain any connectors or operators that depend on
>> internal API) to 1.6.x job manager?/
>>
>> Probably yes, but generally there's no guarantee.
>>
>> The JobGraph is not a public API and thus may change in ways that could
>> cause serialization failures if the client and server versions do not
>> match.
>>
>> On 24.07.2018 17:12, Thomas Weise wrote:
>>> I would expect more external dependencies on the REST API and
>>> suggest to
>>> consider versioning and compatibility guarantees.
>>>
>>> The particular issue is from Beam and tracked here:
>>> https://issues.apache.org/jira/browse/BEAM-4842
>>>
>>> I would be surprised if that is the only instance besides Flink CLI
>>> trying
>>> to use the REST API for job submission. In the case of Beam, it is
>>> problematic to assume that the job service that contains the Flink
>>> client
>>> will always be upgraded lock step with the server.
>>>
>>> What other factors prevent interoperability between Flink versions?
>>> Will it
>>> be possible to submit a job graph built by 1.5.1 client (that does not
>>> contain any connectors or operators that depend on internal API) to
>>> 1.6.x
>>> job manager?
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>> On Tue, Jul 24, 2018 at 7:58 AM Chesnay Schepler
>>> <[hidden email]> wrote:
>>>
>>>> Effectively there are no guarantees for backwards compatibility,
>>>> although we try to maintain it.
>>>>
>>>> There are plans to version the REST API
>>>> (https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
>>>> timeline.
>>>>
>>>> The change you mentioned in particular primarily affects internal code
>>>> (since the removed endpoint is usually only used by the flink CLI).
>>>> Note we do not provide any guarantee that Flink components interact
>>>> properly if the versions don't exactly match, so you should always
>>>> upgrade
>>>> the clients anyway.
>>>>
>>>> On 24.07.2018 16:43, Thomas Weise wrote:
>>>>> Hi,
>>>>>
>>>>> Are there any compatibility guarantees for the Flink REST API or
>>>>> plans to
>>>>> establish the same?
>>>>>
>>>>> I'm asking since I noticed that the 1.5.1 release has an incompatible
>>>>> change that prevents a 1.5.0 client to submit a job to 1.5.0 server
>>>>> (FLINK-9280, removed endpoint /blobserver/port).
>>>>>
>>>>> Such change is especially unexpected in a patch release, but even
>>>>> if it
>>>> was
>>>>> in 1.6.x, it is still problematic because it forces clients to
>>>>> upgrade.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Till Rohrmann
Thanks for bringing this issue up Thomas. I think you've raised a very
important point that we should try to provide not only for the programming
API but also for the REST API compatibility guarantees. Otherwise it is
really hard for 3rd party applications to integrate with Flink.

I think first as a very important step we should try to finalize the
versioning of the REST API as Chesnay suggested. Next we could add another
check list item to the PR template which specifically says that REST
changes need a thorough review. Ideally we could automate the compatibility
check but I'm not aware of a suitable tool for it. Has anyone experience
with any relevant tools?

Cheers,
Till

On Tue, Jul 24, 2018 at 5:53 PM Chesnay Schepler <[hidden email]> wrote:

> Looked some more into it, as of _right now_ a 1.5.X client will not work
> against 1.6.0 cluster
> since the submission body was extended. But we could add a guard against
> this.
>
> On 24.07.2018 17:27, Chesnay Schepler wrote:
> > Note that the changes made to the REST API will in the long-term
> > reduce friction between versions,
> > as we no longer have to submit data through the blob service, which is
> > another internal API.
> >
> > On 24.07.2018 17:25, Chesnay Schepler wrote:
> >> /Will it be possible to submit a job graph built by 1.5.1 client
> >> (that does not contain any connectors or operators that depend on
> >> internal API) to 1.6.x job manager?/
> >>
> >> Probably yes, but generally there's no guarantee.
> >>
> >> The JobGraph is not a public API and thus may change in ways that could
> >> cause serialization failures if the client and server versions do not
> >> match.
> >>
> >> On 24.07.2018 17:12, Thomas Weise wrote:
> >>> I would expect more external dependencies on the REST API and
> >>> suggest to
> >>> consider versioning and compatibility guarantees.
> >>>
> >>> The particular issue is from Beam and tracked here:
> >>> https://issues.apache.org/jira/browse/BEAM-4842
> >>>
> >>> I would be surprised if that is the only instance besides Flink CLI
> >>> trying
> >>> to use the REST API for job submission. In the case of Beam, it is
> >>> problematic to assume that the job service that contains the Flink
> >>> client
> >>> will always be upgraded lock step with the server.
> >>>
> >>> What other factors prevent interoperability between Flink versions?
> >>> Will it
> >>> be possible to submit a job graph built by 1.5.1 client (that does not
> >>> contain any connectors or operators that depend on internal API) to
> >>> 1.6.x
> >>> job manager?
> >>>
> >>> Thanks,
> >>> Thomas
> >>>
> >>>
> >>> On Tue, Jul 24, 2018 at 7:58 AM Chesnay Schepler
> >>> <[hidden email]> wrote:
> >>>
> >>>> Effectively there are no guarantees for backwards compatibility,
> >>>> although we try to maintain it.
> >>>>
> >>>> There are plans to version the REST API
> >>>> (https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
> >>>> timeline.
> >>>>
> >>>> The change you mentioned in particular primarily affects internal code
> >>>> (since the removed endpoint is usually only used by the flink CLI).
> >>>> Note we do not provide any guarantee that Flink components interact
> >>>> properly if the versions don't exactly match, so you should always
> >>>> upgrade
> >>>> the clients anyway.
> >>>>
> >>>> On 24.07.2018 16:43, Thomas Weise wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Are there any compatibility guarantees for the Flink REST API or
> >>>>> plans to
> >>>>> establish the same?
> >>>>>
> >>>>> I'm asking since I noticed that the 1.5.1 release has an incompatible
> >>>>> change that prevents a 1.5.0 client to submit a job to 1.5.0 server
> >>>>> (FLINK-9280, removed endpoint /blobserver/port).
> >>>>>
> >>>>> Such change is especially unexpected in a patch release, but even
> >>>>> if it
> >>>> was
> >>>>> in 1.6.x, it is still problematic because it forces clients to
> >>>>> upgrade.
> >>>>>
> >>>>> Thanks,
> >>>>> Thomas
> >>>>>
> >>>>
> >>
> >>
> >
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Backwards compatibility for REST API

Chesnay Schepler-3
Pretty sure we can piggyback the compatibility test on top of the REST
API E2E test once that is done.
Determine available requests for version A, submit against version B,
see what happens.

On 25.07.2018 09:54, Till Rohrmann wrote:

> Thanks for bringing this issue up Thomas. I think you've raised a very
> important point that we should try to provide not only for the programming
> API but also for the REST API compatibility guarantees. Otherwise it is
> really hard for 3rd party applications to integrate with Flink.
>
> I think first as a very important step we should try to finalize the
> versioning of the REST API as Chesnay suggested. Next we could add another
> check list item to the PR template which specifically says that REST
> changes need a thorough review. Ideally we could automate the compatibility
> check but I'm not aware of a suitable tool for it. Has anyone experience
> with any relevant tools?
>
> Cheers,
> Till
>
> On Tue, Jul 24, 2018 at 5:53 PM Chesnay Schepler <[hidden email]> wrote:
>
>> Looked some more into it, as of _right now_ a 1.5.X client will not work
>> against 1.6.0 cluster
>> since the submission body was extended. But we could add a guard against
>> this.
>>
>> On 24.07.2018 17:27, Chesnay Schepler wrote:
>>> Note that the changes made to the REST API will in the long-term
>>> reduce friction between versions,
>>> as we no longer have to submit data through the blob service, which is
>>> another internal API.
>>>
>>> On 24.07.2018 17:25, Chesnay Schepler wrote:
>>>> /Will it be possible to submit a job graph built by 1.5.1 client
>>>> (that does not contain any connectors or operators that depend on
>>>> internal API) to 1.6.x job manager?/
>>>>
>>>> Probably yes, but generally there's no guarantee.
>>>>
>>>> The JobGraph is not a public API and thus may change in ways that could
>>>> cause serialization failures if the client and server versions do not
>>>> match.
>>>>
>>>> On 24.07.2018 17:12, Thomas Weise wrote:
>>>>> I would expect more external dependencies on the REST API and
>>>>> suggest to
>>>>> consider versioning and compatibility guarantees.
>>>>>
>>>>> The particular issue is from Beam and tracked here:
>>>>> https://issues.apache.org/jira/browse/BEAM-4842
>>>>>
>>>>> I would be surprised if that is the only instance besides Flink CLI
>>>>> trying
>>>>> to use the REST API for job submission. In the case of Beam, it is
>>>>> problematic to assume that the job service that contains the Flink
>>>>> client
>>>>> will always be upgraded lock step with the server.
>>>>>
>>>>> What other factors prevent interoperability between Flink versions?
>>>>> Will it
>>>>> be possible to submit a job graph built by 1.5.1 client (that does not
>>>>> contain any connectors or operators that depend on internal API) to
>>>>> 1.6.x
>>>>> job manager?
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Tue, Jul 24, 2018 at 7:58 AM Chesnay Schepler
>>>>> <[hidden email]> wrote:
>>>>>
>>>>>> Effectively there are no guarantees for backwards compatibility,
>>>>>> although we try to maintain it.
>>>>>>
>>>>>> There are plans to version the REST API
>>>>>> (https://issues.apache.org/jira/browse/FLINK-7551) but no definitive
>>>>>> timeline.
>>>>>>
>>>>>> The change you mentioned in particular primarily affects internal code
>>>>>> (since the removed endpoint is usually only used by the flink CLI).
>>>>>> Note we do not provide any guarantee that Flink components interact
>>>>>> properly if the versions don't exactly match, so you should always
>>>>>> upgrade
>>>>>> the clients anyway.
>>>>>>
>>>>>> On 24.07.2018 16:43, Thomas Weise wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Are there any compatibility guarantees for the Flink REST API or
>>>>>>> plans to
>>>>>>> establish the same?
>>>>>>>
>>>>>>> I'm asking since I noticed that the 1.5.1 release has an incompatible
>>>>>>> change that prevents a 1.5.0 client to submit a job to 1.5.0 server
>>>>>>> (FLINK-9280, removed endpoint /blobserver/port).
>>>>>>>
>>>>>>> Such change is especially unexpected in a patch release, but even
>>>>>>> if it
>>>>>> was
>>>>>>> in 1.6.x, it is still problematic because it forces clients to
>>>>>>> upgrade.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>
>>>
>>