[DISCUSS] Dealing with deprecated and legacy code in Flink

[Timo Walther-2]

Hi everyone,

I would like to start a discussion how we treat deprecated and legacy
code in Flink in the future. During the last years, our code base has
grown quite a bit and a couple of interfaces and components have been
reworked on the way.

I'm sure each component has a few legacy parts that are waiting for
removal. Apart from keeping outdated API around for a couple of releases
until users have updated their code, it is also often easier to just put
a @Deprecation annotation and postpone the actual change.

When looking at the current code, we have duplicated SQL planners,
duplicated APIs (DataSet/DataStream), duplicated source/sink interfaces,
outdated connectors (Elasticsearch 5?) and dependencies (Scala 2.11?).

I'm wondering whether we should come up with some legacy/deprecation
guidelines for the future.

Some examples:

- I could imagine new Flink-specific annotations for documenting (in
code) in which version an interface was deprecated and when the planned
removal should take place.
- Or guidelines that we drop a connector when the external project does
not maintain the version for 6 months etc.

Plannable removal dates should also help users to not be surprised when
a connector or Scala version is not supported anymore.

What do you think? I'm very happy to hear more opinions.

Regards,
Timo

[Piotr Nowojski-5]

Hi Timo,

Thanks for starting this discussion. I'm not sure how we should approach
this topic and what should be our final recommendation, but definitely
clearing up a couple of things would be helpful.

For starters, I agree it would be good to have some more information,
besides just "@Deprecated" annotations. Is it possible to extend
annotations with informations like:
- from which version was it deprecated
- when is it planned to be removed (we could always mark `2.0` as "never"
;) )
- add maybe some pre/post release step of verifying that removal has
actually happened?

?

On the other hand, I think it's very important to maintain backward
compatibility with Flink as much as possible. As a developer I don't
like dealing with this, but as a user I hate dealing with incompatible
upgrades even more. So all in all, I would be in favour of putting more
effort not in deprecating and removing APIs, but making sure that they are
stable.

Stephan Ewan also raised a point sometime ago, that in the recent past, we
developed a habit of marking everything as `@Experimental` or
`@PublicEvolving` and leaving it as that forever. Maybe we should also
include deadlines (2 releases since introduction?) for changing
`@Experimental`/`@PublicEvolving` into `@Public` in this kind of
guidelines/automated checks?

Piotrek

pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]> napisał(a):

> Hi everyone,
>
> I would like to start a discussion how we treat deprecated and legacy
> code in Flink in the future. During the last years, our code base has
> grown quite a bit and a couple of interfaces and components have been
> reworked on the way.
>
> I'm sure each component has a few legacy parts that are waiting for
> removal. Apart from keeping outdated API around for a couple of releases
> until users have updated their code, it is also often easier to just put
> a @Deprecation annotation and postpone the actual change.
>
> When looking at the current code, we have duplicated SQL planners,
> duplicated APIs (DataSet/DataStream), duplicated source/sink interfaces,
> outdated connectors (Elasticsearch 5?) and dependencies (Scala 2.11?).
>
> I'm wondering whether we should come up with some legacy/deprecation
> guidelines for the future.
>
> Some examples:
>
> - I could imagine new Flink-specific annotations for documenting (in
> code) in which version an interface was deprecated and when the planned
> removal should take place.
> - Or guidelines that we drop a connector when the external project does
> not maintain the version for 6 months etc.
>
> Plannable removal dates should also help users to not be surprised when
> a connector or Scala version is not supported anymore.
>
> What do you think? I'm very happy to hear more opinions.
>
> Regards,
> Timo
>
>
>
>
>
>

[Yun Gao]

Hi,

Very thanks for @Timo to initiate the discussion!

I would also +1 for providing some informations to users via annotations
or documents in advanced to not suprise users before we actually remove the legacy code.
If we finally decide to change one functionality that user could sense, perhaps one
premise is that Flink has provided a replacement for that one and users could transfer their
applications easily. Then we might also consider have one dedicated documentation page
to list the functionalities to change and how users could do the transfer.

To make the decision of whether to remove some legacy code, we might also consider to have a survey
like the one we did for mesos support [1] to see how this functionality is used.

Best,
 Yun


[1] https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E



 ------------------Original Mail ------------------
Sender:Piotr Nowojski <[hidden email]>
Send Date:Mon Jan 18 18:23:36 2021
Recipients:dev <[hidden email]>
Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in Flink
Hi Timo,

Thanks for starting this discussion. I'm not sure how we should approach
this topic and what should be our final recommendation, but definitely
clearing up a couple of things would be helpful.

For starters, I agree it would be good to have some more information,
besides just "@Deprecated" annotations. Is it possible to extend
annotations with informations like:
- from which version was it deprecated
- when is it planned to be removed (we could always mark `2.0` as "never"
;) )
- add maybe some pre/post release step of verifying that removal has
actually happened?

?

On the other hand, I think it's very important to maintain backward
compatibility with Flink as much as possible. As a developer I don't
like dealing with this, but as a user I hate dealing with incompatible
upgrades even more. So all in all, I would be in favour of putting more
effort not in deprecating and removing APIs, but making sure that they are
stable.

Stephan Ewan also raised a point sometime ago, that in the recent past, we
developed a habit of marking everything as `@Experimental` or
`@PublicEvolving` and leaving it as that forever. Maybe we should also
include deadlines (2 releases since introduction?) for changing
`@Experimental`/`@PublicEvolving` into `@Public` in this kind of
guidelines/automated checks?

Piotrek

pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]> napisał(a):

> Hi everyone,
>
> I would like to start a discussion how we treat deprecated and legacy
> code in Flink in the future. During the last years, our code base has
> grown quite a bit and a couple of interfaces and components have been
> reworked on the way.
>
> I'm sure each component has a few legacy parts that are waiting for
> removal. Apart from keeping outdated API around for a couple of releases
> until users have updated their code, it is also often easier to just put
> a @Deprecation annotation and postpone the actual change.
>
> When looking at the current code, we have duplicated SQL planners,
> duplicated APIs (DataSet/DataStream), duplicated source/sink interfaces,
> outdated connectors (Elasticsearch 5?) and dependencies (Scala 2.11?).
>
> I'm wondering whether we should come up with some legacy/deprecation
> guidelines for the future.
>
> Some examples:
>
> - I could imagine new Flink-specific annotations for documenting (in
> code) in which version an interface was deprecated and when the planned
> removal should take place.
> - Or guidelines that we drop a connector when the external project does
> not maintain the version for 6 months etc.
>
> Plannable removal dates should also help users to not be surprised when
> a connector or Scala version is not supported anymore.
>
> What do you think? I'm very happy to hear more opinions.
>
> Regards,
> Timo
>
>
>
>
>
>

[Till Rohrmann]

Thanks a lot for starting this discussion Timo. I like the idea of setting
more explicit guidelines for deprecating functionality.

I really like the idea of adding with the @Deprecated annotation since when
the function is deprecated. Based on that one can simply search for
features which should be removed in a given release. Alternatively, one
could as you said also state the removal version.

I think what also works is to directly create a critical JIRA issue with
removing functionality as soon as one deprecates something. The problem was
often that after deprecating something, it gets forgotten.

For dropping connectors I am a bit uncertain. From a project management
perspective it sounds like a good idea to not have to support connectors
which are no longer supported for some time. However, what if this
connector is still very popular and in heavy use by our users? Just because
an external system or a version of it is no longer maintained does not mean
that the system is no longer used. I think our current practice with trying
to judge whether our users still use this feature/connector works somewhat.
On the other hand, having these guidelines would probably make it easier to
argue to remove something even if there are still a couple of users.

Cheers,
Till

On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]>
wrote:

> Hi,
>
> Very thanks for @Timo to initiate the discussion!
>
> I would also +1 for providing some informations to users via annotations
> or documents in advanced to not suprise users before we actually remove
> the legacy code.
> If we finally decide to change one functionality that user could sense,
> perhaps one
> premise is that Flink has provided a replacement for that one and users
> could transfer their
> applications easily. Then we might also consider have one dedicated
> documentation page
> to list the functionalities to change and how users could do the transfer.
>
> To make the decision of whether to remove some legacy code, we might also
> consider to have a survey
> like the one we did for mesos support [1] to see how this functionality is
> used.
>
> Best,
>  Yun
>
>
> [1]
> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
>
>
>
>  ------------------Original Mail ------------------
> Sender:Piotr Nowojski <[hidden email]>
> Send Date:Mon Jan 18 18:23:36 2021
> Recipients:dev <[hidden email]>
> Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in Flink
> Hi Timo,
>
> Thanks for starting this discussion. I'm not sure how we should approach
> this topic and what should be our final recommendation, but definitely
> clearing up a couple of things would be helpful.
>
> For starters, I agree it would be good to have some more information,
> besides just "@Deprecated" annotations. Is it possible to extend
> annotations with informations like:
> - from which version was it deprecated
> - when is it planned to be removed (we could always mark `2.0` as "never"
> ;) )
> - add maybe some pre/post release step of verifying that removal has
> actually happened?
>
> ?
>
> On the other hand, I think it's very important to maintain backward
> compatibility with Flink as much as possible. As a developer I don't
> like dealing with this, but as a user I hate dealing with incompatible
> upgrades even more. So all in all, I would be in favour of putting more
> effort not in deprecating and removing APIs, but making sure that they are
> stable.
>
> Stephan Ewan also raised a point sometime ago, that in the recent past, we
> developed a habit of marking everything as `@Experimental` or
> `@PublicEvolving` and leaving it as that forever. Maybe we should also
> include deadlines (2 releases since introduction?) for changing
> `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
> guidelines/automated checks?
>
> Piotrek
>
> pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]> napisał(a):
>
> > Hi everyone,
> >
> > I would like to start a discussion how we treat deprecated and legacy
> > code in Flink in the future. During the last years, our code base has
> > grown quite a bit and a couple of interfaces and components have been
> > reworked on the way.
> >
> > I'm sure each component has a few legacy parts that are waiting for
> > removal. Apart from keeping outdated API around for a couple of releases
> > until users have updated their code, it is also often easier to just put
> > a @Deprecation annotation and postpone the actual change.
> >
> > When looking at the current code, we have duplicated SQL planners,
> > duplicated APIs (DataSet/DataStream), duplicated source/sink interfaces,
> > outdated connectors (Elasticsearch 5?) and dependencies (Scala 2.11?).
> >
> > I'm wondering whether we should come up with some legacy/deprecation
> > guidelines for the future.
> >
> > Some examples:
> >
> > - I could imagine new Flink-specific annotations for documenting (in
> > code) in which version an interface was deprecated and when the planned
> > removal should take place.
> > - Or guidelines that we drop a connector when the external project does
> > not maintain the version for 6 months etc.
> >
> > Plannable removal dates should also help users to not be surprised when
> > a connector or Scala version is not supported anymore.
> >
> > What do you think? I'm very happy to hear more opinions.
> >
> > Regards,
> > Timo
> >
> >
> >
> >
> >
> >
>

[Matthias]

Thanks Timo for opening this discussion.

+1 I like the idea of adding a deprecation deadline and/or information when
the
functionality was deprecated. It looks like this is already done in the
PyFlink code.

Creating a JIRA issue for removing the functionality, as Till suggested,
might help to
maintain this process of removing the deprecated functionality. I'd prefer
that over
relying on the release manager (assuming that he/she would run the check as
part
of the release process) to identify functionality that should have been
removed as
part of the release. But ok, that might be a team decision.

For the connectors: Can't we assume that users would reach out to us if we
deprecate
a connector assuming that they can conclude that this connector will,
otherwise, disappear.
Maybe, that needs to be mentioned in the deprecation information as well,
then.
This would have the benefit of getting direct feedback about how much a
connector is still in
use and may open the doors for other contributors to offer help like it
happened for the
Mesos support [1].

And about the idea of adding such deadlines to @Public, @PublicEvolving,
and @Experimental:
Is it actually possible to have a fixed timeframe for these annotations to
change? I would
imagine that it depends on the underlying feature how long an API
is @PublicEvolving or
@Experimental? But it sounds still like a good idea to trigger warnings for
those annotations
in case they haven't been touched for a while. Therefore, I would second
this suggestion.

Best,
Matthias

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985

On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <[hidden email]> wrote:

> Thanks a lot for starting this discussion Timo. I like the idea of setting
> more explicit guidelines for deprecating functionality.
>
> I really like the idea of adding with the @Deprecated annotation since when
> the function is deprecated. Based on that one can simply search for
> features which should be removed in a given release. Alternatively, one
> could as you said also state the removal version.
>
> I think what also works is to directly create a critical JIRA issue with
> removing functionality as soon as one deprecates something. The problem was
> often that after deprecating something, it gets forgotten.
>
> For dropping connectors I am a bit uncertain. From a project management
> perspective it sounds like a good idea to not have to support connectors
> which are no longer supported for some time. However, what if this
> connector is still very popular and in heavy use by our users? Just because
> an external system or a version of it is no longer maintained does not mean
> that the system is no longer used. I think our current practice with trying
> to judge whether our users still use this feature/connector works somewhat.
> On the other hand, having these guidelines would probably make it easier to
> argue to remove something even if there are still a couple of users.
>
> Cheers,
> Till
>
> On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Very thanks for @Timo to initiate the discussion!
> >
> > I would also +1 for providing some informations to users via annotations
> > or documents in advanced to not suprise users before we actually remove
> > the legacy code.
> > If we finally decide to change one functionality that user could sense,
> > perhaps one
> > premise is that Flink has provided a replacement for that one and users
> > could transfer their
> > applications easily. Then we might also consider have one dedicated
> > documentation page
> > to list the functionalities to change and how users could do the
> transfer.
> >
> > To make the decision of whether to remove some legacy code, we might also
> > consider to have a survey
> > like the one we did for mesos support [1] to see how this functionality
> is
> > used.
> >
> > Best,
> >  Yun
> >
> >
> > [1]
> >
> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
> >
> >
> >
> >  ------------------Original Mail ------------------
> > Sender:Piotr Nowojski <[hidden email]>
> > Send Date:Mon Jan 18 18:23:36 2021
> > Recipients:dev <[hidden email]>
> > Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in Flink
> > Hi Timo,
> >
> > Thanks for starting this discussion. I'm not sure how we should approach
> > this topic and what should be our final recommendation, but definitely
> > clearing up a couple of things would be helpful.
> >
> > For starters, I agree it would be good to have some more information,
> > besides just "@Deprecated" annotations. Is it possible to extend
> > annotations with informations like:
> > - from which version was it deprecated
> > - when is it planned to be removed (we could always mark `2.0` as "never"
> > ;) )
> > - add maybe some pre/post release step of verifying that removal has
> > actually happened?
> >
> > ?
> >
> > On the other hand, I think it's very important to maintain backward
> > compatibility with Flink as much as possible. As a developer I don't
> > like dealing with this, but as a user I hate dealing with incompatible
> > upgrades even more. So all in all, I would be in favour of putting more
> > effort not in deprecating and removing APIs, but making sure that they
> are
> > stable.
> >
> > Stephan Ewan also raised a point sometime ago, that in the recent past,
> we
> > developed a habit of marking everything as `@Experimental` or
> > `@PublicEvolving` and leaving it as that forever. Maybe we should also
> > include deadlines (2 releases since introduction?) for changing
> > `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
> > guidelines/automated checks?
> >
> > Piotrek
> >
> > pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]> napisał(a):
> >
> > > Hi everyone,
> > >
> > > I would like to start a discussion how we treat deprecated and legacy
> > > code in Flink in the future. During the last years, our code base has
> > > grown quite a bit and a couple of interfaces and components have been
> > > reworked on the way.
> > >
> > > I'm sure each component has a few legacy parts that are waiting for
> > > removal. Apart from keeping outdated API around for a couple of
> releases
> > > until users have updated their code, it is also often easier to just
> put
> > > a @Deprecation annotation and postpone the actual change.
> > >
> > > When looking at the current code, we have duplicated SQL planners,
> > > duplicated APIs (DataSet/DataStream), duplicated source/sink
> interfaces,
> > > outdated connectors (Elasticsearch 5?) and dependencies (Scala 2.11?).
> > >
> > > I'm wondering whether we should come up with some legacy/deprecation
> > > guidelines for the future.
> > >
> > > Some examples:
> > >
> > > - I could imagine new Flink-specific annotations for documenting (in
> > > code) in which version an interface was deprecated and when the planned
> > > removal should take place.
> > > - Or guidelines that we drop a connector when the external project does
> > > not maintain the version for 6 months etc.
> > >
> > > Plannable removal dates should also help users to not be surprised when
> > > a connector or Scala version is not supported anymore.
> > >
> > > What do you think? I'm very happy to hear more opinions.
> > >
> > > Regards,
> > > Timo
> > >
> > >
> > >
> > >
> > >
> > >
> >

[Piotr Nowojski-5]

Hi,

I would prefer not to rely on the Jira for marking when something is
supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)` would
have two obligatory parameters, there would be no way to "forget" about
marking it and it would be also self documenting (I don't imagine users
using JIRA to check this kind of things). We can have Jira tickets for
those things for tracking purposes on the JIRA release board, but relying
only on JIRA tickets I think is just asking for inconsistencies.

> Is it actually possible to have a fixed timeframe for these annotations
to change?
> I would imagine that it depends on the underlying feature how long an API
is @PublicEvolving or @Experimental?

I agree it would depend on the feature, hence different features might have
longer or shorter "unstable" timeframes. But I'm afraid if we won't start
thinking about fixing this timeframe, we would too often end up with
perpetually "unstable" APIs. I don't know where I would draw the line
exactly, but assuming we want to have stable APIs, if something is marked
`@PublicEvolving` or `@Experimental` for 3 years, IMO it should be switched
to `@Public` by default (or be moved out of the main repo?).

Piotrek

śr., 20 sty 2021 o 09:54 Matthias Pohl <[hidden email]> napisał(a):

> Thanks Timo for opening this discussion.
>
> +1 I like the idea of adding a deprecation deadline and/or information when
> the
> functionality was deprecated. It looks like this is already done in the
> PyFlink code.
>
> Creating a JIRA issue for removing the functionality, as Till suggested,
> might help to
> maintain this process of removing the deprecated functionality. I'd prefer
> that over
> relying on the release manager (assuming that he/she would run the check as
> part
> of the release process) to identify functionality that should have been
> removed as
> part of the release. But ok, that might be a team decision.
>
> For the connectors: Can't we assume that users would reach out to us if we
> deprecate
> a connector assuming that they can conclude that this connector will,
> otherwise, disappear.
> Maybe, that needs to be mentioned in the deprecation information as well,
> then.
> This would have the benefit of getting direct feedback about how much a
> connector is still in
> use and may open the doors for other contributors to offer help like it
> happened for the
> Mesos support [1].
>
> And about the idea of adding such deadlines to @Public, @PublicEvolving,
> and @Experimental:
> Is it actually possible to have a fixed timeframe for these annotations to
> change? I would
> imagine that it depends on the underlying feature how long an API
> is @PublicEvolving or
> @Experimental? But it sounds still like a good idea to trigger warnings for
> those annotations
> in case they haven't been touched for a while. Therefore, I would second
> this suggestion.
>
> Best,
> Matthias
>
> [1]
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985
>
> On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <[hidden email]>
> wrote:
>
> > Thanks a lot for starting this discussion Timo. I like the idea of
> setting
> > more explicit guidelines for deprecating functionality.
> >
> > I really like the idea of adding with the @Deprecated annotation since
> when
> > the function is deprecated. Based on that one can simply search for
> > features which should be removed in a given release. Alternatively, one
> > could as you said also state the removal version.
> >
> > I think what also works is to directly create a critical JIRA issue with
> > removing functionality as soon as one deprecates something. The problem
> was
> > often that after deprecating something, it gets forgotten.
> >
> > For dropping connectors I am a bit uncertain. From a project management
> > perspective it sounds like a good idea to not have to support connectors
> > which are no longer supported for some time. However, what if this
> > connector is still very popular and in heavy use by our users? Just
> because
> > an external system or a version of it is no longer maintained does not
> mean
> > that the system is no longer used. I think our current practice with
> trying
> > to judge whether our users still use this feature/connector works
> somewhat.
> > On the other hand, having these guidelines would probably make it easier
> to
> > argue to remove something even if there are still a couple of users.
> >
> > Cheers,
> > Till
> >
> > On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]>
> > wrote:
> >
> > > Hi,
> > >
> > > Very thanks for @Timo to initiate the discussion!
> > >
> > > I would also +1 for providing some informations to users via
> annotations
> > > or documents in advanced to not suprise users before we actually remove
> > > the legacy code.
> > > If we finally decide to change one functionality that user could sense,
> > > perhaps one
> > > premise is that Flink has provided a replacement for that one and users
> > > could transfer their
> > > applications easily. Then we might also consider have one dedicated
> > > documentation page
> > > to list the functionalities to change and how users could do the
> > transfer.
> > >
> > > To make the decision of whether to remove some legacy code, we might
> also
> > > consider to have a survey
> > > like the one we did for mesos support [1] to see how this functionality
> > is
> > > used.
> > >
> > > Best,
> > >  Yun
> > >
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
> > >
> > >
> > >
> > >  ------------------Original Mail ------------------
> > > Sender:Piotr Nowojski <[hidden email]>
> > > Send Date:Mon Jan 18 18:23:36 2021
> > > Recipients:dev <[hidden email]>
> > > Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in Flink
> > > Hi Timo,
> > >
> > > Thanks for starting this discussion. I'm not sure how we should
> approach
> > > this topic and what should be our final recommendation, but definitely
> > > clearing up a couple of things would be helpful.
> > >
> > > For starters, I agree it would be good to have some more information,
> > > besides just "@Deprecated" annotations. Is it possible to extend
> > > annotations with informations like:
> > > - from which version was it deprecated
> > > - when is it planned to be removed (we could always mark `2.0` as
> "never"
> > > ;) )
> > > - add maybe some pre/post release step of verifying that removal has
> > > actually happened?
> > >
> > > ?
> > >
> > > On the other hand, I think it's very important to maintain backward
> > > compatibility with Flink as much as possible. As a developer I don't
> > > like dealing with this, but as a user I hate dealing with incompatible
> > > upgrades even more. So all in all, I would be in favour of putting more
> > > effort not in deprecating and removing APIs, but making sure that they
> > are
> > > stable.
> > >
> > > Stephan Ewan also raised a point sometime ago, that in the recent past,
> > we
> > > developed a habit of marking everything as `@Experimental` or
> > > `@PublicEvolving` and leaving it as that forever. Maybe we should also
> > > include deadlines (2 releases since introduction?) for changing
> > > `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
> > > guidelines/automated checks?
> > >
> > > Piotrek
> > >
> > > pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]> napisał(a):
> > >
> > > > Hi everyone,
> > > >
> > > > I would like to start a discussion how we treat deprecated and legacy
> > > > code in Flink in the future. During the last years, our code base has
> > > > grown quite a bit and a couple of interfaces and components have been
> > > > reworked on the way.
> > > >
> > > > I'm sure each component has a few legacy parts that are waiting for
> > > > removal. Apart from keeping outdated API around for a couple of
> > releases
> > > > until users have updated their code, it is also often easier to just
> > put
> > > > a @Deprecation annotation and postpone the actual change.
> > > >
> > > > When looking at the current code, we have duplicated SQL planners,
> > > > duplicated APIs (DataSet/DataStream), duplicated source/sink
> > interfaces,
> > > > outdated connectors (Elasticsearch 5?) and dependencies (Scala
> 2.11?).
> > > >
> > > > I'm wondering whether we should come up with some legacy/deprecation
> > > > guidelines for the future.
> > > >
> > > > Some examples:
> > > >
> > > > - I could imagine new Flink-specific annotations for documenting (in
> > > > code) in which version an interface was deprecated and when the
> planned
> > > > removal should take place.
> > > > - Or guidelines that we drop a connector when the external project
> does
> > > > not maintain the version for 6 months etc.
> > > >
> > > > Plannable removal dates should also help users to not be surprised
> when
> > > > a connector or Scala version is not supported anymore.
> > > >
> > > > What do you think? I'm very happy to hear more opinions.
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
>

[Khachatryan Roman]

Hi,

I think having two Deprecated annotations (Flink and Java) may be confusing.
One alternative is to combine standard annotation with mandatory Javadocs
tags (checked with checkstyle).
And starting from Java 9 it has "since" and "forRemoval" arguments.


Regards,
Roman


On Wed, Jan 20, 2021 at 2:01 PM Piotr Nowojski <[hidden email]> wrote:

> Hi,
>
> I would prefer not to rely on the Jira for marking when something is
> supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)` would
> have two obligatory parameters, there would be no way to "forget" about
> marking it and it would be also self documenting (I don't imagine users
> using JIRA to check this kind of things). We can have Jira tickets for
> those things for tracking purposes on the JIRA release board, but relying
> only on JIRA tickets I think is just asking for inconsistencies.
>
> > Is it actually possible to have a fixed timeframe for these annotations
> to change?
> > I would imagine that it depends on the underlying feature how long an API
> is @PublicEvolving or @Experimental?
>
> I agree it would depend on the feature, hence different features might have
> longer or shorter "unstable" timeframes. But I'm afraid if we won't start
> thinking about fixing this timeframe, we would too often end up with
> perpetually "unstable" APIs. I don't know where I would draw the line
> exactly, but assuming we want to have stable APIs, if something is marked
> `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be switched
> to `@Public` by default (or be moved out of the main repo?).
>
> Piotrek
>
> śr., 20 sty 2021 o 09:54 Matthias Pohl <[hidden email]>
> napisał(a):
>
> > Thanks Timo for opening this discussion.
> >
> > +1 I like the idea of adding a deprecation deadline and/or information
> when
> > the
> > functionality was deprecated. It looks like this is already done in the
> > PyFlink code.
> >
> > Creating a JIRA issue for removing the functionality, as Till suggested,
> > might help to
> > maintain this process of removing the deprecated functionality. I'd
> prefer
> > that over
> > relying on the release manager (assuming that he/she would run the check
> as
> > part
> > of the release process) to identify functionality that should have been
> > removed as
> > part of the release. But ok, that might be a team decision.
> >
> > For the connectors: Can't we assume that users would reach out to us if
> we
> > deprecate
> > a connector assuming that they can conclude that this connector will,
> > otherwise, disappear.
> > Maybe, that needs to be mentioned in the deprecation information as well,
> > then.
> > This would have the benefit of getting direct feedback about how much a
> > connector is still in
> > use and may open the doors for other contributors to offer help like it
> > happened for the
> > Mesos support [1].
> >
> > And about the idea of adding such deadlines to @Public, @PublicEvolving,
> > and @Experimental:
> > Is it actually possible to have a fixed timeframe for these annotations
> to
> > change? I would
> > imagine that it depends on the underlying feature how long an API
> > is @PublicEvolving or
> > @Experimental? But it sounds still like a good idea to trigger warnings
> for
> > those annotations
> > in case they haven't been touched for a while. Therefore, I would second
> > this suggestion.
> >
> > Best,
> > Matthias
> >
> > [1]
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985
> >
> > On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <[hidden email]>
> > wrote:
> >
> > > Thanks a lot for starting this discussion Timo. I like the idea of
> > setting
> > > more explicit guidelines for deprecating functionality.
> > >
> > > I really like the idea of adding with the @Deprecated annotation since
> > when
> > > the function is deprecated. Based on that one can simply search for
> > > features which should be removed in a given release. Alternatively, one
> > > could as you said also state the removal version.
> > >
> > > I think what also works is to directly create a critical JIRA issue
> with
> > > removing functionality as soon as one deprecates something. The problem
> > was
> > > often that after deprecating something, it gets forgotten.
> > >
> > > For dropping connectors I am a bit uncertain. From a project management
> > > perspective it sounds like a good idea to not have to support
> connectors
> > > which are no longer supported for some time. However, what if this
> > > connector is still very popular and in heavy use by our users? Just
> > because
> > > an external system or a version of it is no longer maintained does not
> > mean
> > > that the system is no longer used. I think our current practice with
> > trying
> > > to judge whether our users still use this feature/connector works
> > somewhat.
> > > On the other hand, having these guidelines would probably make it
> easier
> > to
> > > argue to remove something even if there are still a couple of users.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Very thanks for @Timo to initiate the discussion!
> > > >
> > > > I would also +1 for providing some informations to users via
> > annotations
> > > > or documents in advanced to not suprise users before we actually
> remove
> > > > the legacy code.
> > > > If we finally decide to change one functionality that user could
> sense,
> > > > perhaps one
> > > > premise is that Flink has provided a replacement for that one and
> users
> > > > could transfer their
> > > > applications easily. Then we might also consider have one dedicated
> > > > documentation page
> > > > to list the functionalities to change and how users could do the
> > > transfer.
> > > >
> > > > To make the decision of whether to remove some legacy code, we might
> > also
> > > > consider to have a survey
> > > > like the one we did for mesos support [1] to see how this
> functionality
> > > is
> > > > used.
> > > >
> > > > Best,
> > > >  Yun
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
> > > >
> > > >
> > > >
> > > >  ------------------Original Mail ------------------
> > > > Sender:Piotr Nowojski <[hidden email]>
> > > > Send Date:Mon Jan 18 18:23:36 2021
> > > > Recipients:dev <[hidden email]>
> > > > Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in
> Flink
> > > > Hi Timo,
> > > >
> > > > Thanks for starting this discussion. I'm not sure how we should
> > approach
> > > > this topic and what should be our final recommendation, but
> definitely
> > > > clearing up a couple of things would be helpful.
> > > >
> > > > For starters, I agree it would be good to have some more information,
> > > > besides just "@Deprecated" annotations. Is it possible to extend
> > > > annotations with informations like:
> > > > - from which version was it deprecated
> > > > - when is it planned to be removed (we could always mark `2.0` as
> > "never"
> > > > ;) )
> > > > - add maybe some pre/post release step of verifying that removal has
> > > > actually happened?
> > > >
> > > > ?
> > > >
> > > > On the other hand, I think it's very important to maintain backward
> > > > compatibility with Flink as much as possible. As a developer I don't
> > > > like dealing with this, but as a user I hate dealing with
> incompatible
> > > > upgrades even more. So all in all, I would be in favour of putting
> more
> > > > effort not in deprecating and removing APIs, but making sure that
> they
> > > are
> > > > stable.
> > > >
> > > > Stephan Ewan also raised a point sometime ago, that in the recent
> past,
> > > we
> > > > developed a habit of marking everything as `@Experimental` or
> > > > `@PublicEvolving` and leaving it as that forever. Maybe we should
> also
> > > > include deadlines (2 releases since introduction?) for changing
> > > > `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
> > > > guidelines/automated checks?
> > > >
> > > > Piotrek
> > > >
> > > > pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]>
> napisał(a):
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I would like to start a discussion how we treat deprecated and
> legacy
> > > > > code in Flink in the future. During the last years, our code base
> has
> > > > > grown quite a bit and a couple of interfaces and components have
> been
> > > > > reworked on the way.
> > > > >
> > > > > I'm sure each component has a few legacy parts that are waiting for
> > > > > removal. Apart from keeping outdated API around for a couple of
> > > releases
> > > > > until users have updated their code, it is also often easier to
> just
> > > put
> > > > > a @Deprecation annotation and postpone the actual change.
> > > > >
> > > > > When looking at the current code, we have duplicated SQL planners,
> > > > > duplicated APIs (DataSet/DataStream), duplicated source/sink
> > > interfaces,
> > > > > outdated connectors (Elasticsearch 5?) and dependencies (Scala
> > 2.11?).
> > > > >
> > > > > I'm wondering whether we should come up with some
> legacy/deprecation
> > > > > guidelines for the future.
> > > > >
> > > > > Some examples:
> > > > >
> > > > > - I could imagine new Flink-specific annotations for documenting
> (in
> > > > > code) in which version an interface was deprecated and when the
> > planned
> > > > > removal should take place.
> > > > > - Or guidelines that we drop a connector when the external project
> > does
> > > > > not maintain the version for 6 months etc.
> > > > >
> > > > > Plannable removal dates should also help users to not be surprised
> > when
> > > > > a connector or Scala version is not supported anymore.
> > > > >
> > > > > What do you think? I'm very happy to hear more opinions.
> > > > >
> > > > > Regards,
> > > > > Timo
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> >
>

[Matthias]

>
> I would prefer not to rely on the Jira for marking when something is
> supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)` would
> have two obligatory parameters, there would be no way to "forget" about
> marking it and it would be also self documenting (I don't imagine users
> using JIRA to check this kind of things). We can have Jira tickets for
> those things for tracking purposes on the JIRA release board, but relying
> only on JIRA tickets I think is just asking for inconsistencies.


Yes, I totally agree with you. The annotation should be the way to document
it. I just saw the Jira issue proposal as a good way to track these things
within the team for code that is not visited regularly. But as I said, that
depends on how each team organizes itself. I don't see this as a process
everyone needs to follow.

I agree it would depend on the feature, hence different features might have
> longer or shorter "unstable" timeframes. But I'm afraid if we won't start
> thinking about fixing this timeframe, we would too often end up with
> perpetually "unstable" APIs. I don't know where I would draw the line
> exactly, but assuming we want to have stable APIs, if something is marked
> `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be switched
> to `@Public` by default (or be moved out of the main repo?).


We could use the timestamp for the @PublicEvolving and @Experimental as an
indicator to revisit the annotation. Let's say, we revisit each annotation
after a specific amount of time (e.g. after two releases) to discuss
whether the API is still evolving or actually stable. If it's stable, we
change the annotation to @Public. If it's not stable, we update the
timestamp.

Best,
Matthias

On Wed, Jan 20, 2021 at 2:01 PM Piotr Nowojski <[hidden email]> wrote:

> Hi,
>
> I would prefer not to rely on the Jira for marking when something is
> supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)` would
> have two obligatory parameters, there would be no way to "forget" about
> marking it and it would be also self documenting (I don't imagine users
> using JIRA to check this kind of things). We can have Jira tickets for
> those things for tracking purposes on the JIRA release board, but relying
> only on JIRA tickets I think is just asking for inconsistencies.
>
> > Is it actually possible to have a fixed timeframe for these annotations
> to change?
> > I would imagine that it depends on the underlying feature how long an
> API is @PublicEvolving or @Experimental?
>
> I agree it would depend on the feature, hence different features might
> have longer or shorter "unstable" timeframes. But I'm afraid if we won't
> start thinking about fixing this timeframe, we would too often end up with
> perpetually "unstable" APIs. I don't know where I would draw the line
> exactly, but assuming we want to have stable APIs, if something is marked
> `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be switched
> to `@Public` by default (or be moved out of the main repo?).
>
> Piotrek
>
> śr., 20 sty 2021 o 09:54 Matthias Pohl <[hidden email]>
> napisał(a):
>
>> Thanks Timo for opening this discussion.
>>
>> +1 I like the idea of adding a deprecation deadline and/or information
>> when
>> the
>> functionality was deprecated. It looks like this is already done in the
>> PyFlink code.
>>
>> Creating a JIRA issue for removing the functionality, as Till suggested,
>> might help to
>> maintain this process of removing the deprecated functionality. I'd prefer
>> that over
>> relying on the release manager (assuming that he/she would run the check
>> as
>> part
>> of the release process) to identify functionality that should have been
>> removed as
>> part of the release. But ok, that might be a team decision.
>>
>> For the connectors: Can't we assume that users would reach out to us if we
>> deprecate
>> a connector assuming that they can conclude that this connector will,
>> otherwise, disappear.
>> Maybe, that needs to be mentioned in the deprecation information as well,
>> then.
>> This would have the benefit of getting direct feedback about how much a
>> connector is still in
>> use and may open the doors for other contributors to offer help like it
>> happened for the
>> Mesos support [1].
>>
>> And about the idea of adding such deadlines to @Public, @PublicEvolving,
>> and @Experimental:
>> Is it actually possible to have a fixed timeframe for these annotations to
>> change? I would
>> imagine that it depends on the underlying feature how long an API
>> is @PublicEvolving or
>> @Experimental? But it sounds still like a good idea to trigger warnings
>> for
>> those annotations
>> in case they haven't been touched for a while. Therefore, I would second
>> this suggestion.
>>
>> Best,
>> Matthias
>>
>> [1]
>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985
>>
>> On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <[hidden email]>
>> wrote:
>>
>> > Thanks a lot for starting this discussion Timo. I like the idea of
>> setting
>> > more explicit guidelines for deprecating functionality.
>> >
>> > I really like the idea of adding with the @Deprecated annotation since
>> when
>> > the function is deprecated. Based on that one can simply search for
>> > features which should be removed in a given release. Alternatively, one
>> > could as you said also state the removal version.
>> >
>> > I think what also works is to directly create a critical JIRA issue with
>> > removing functionality as soon as one deprecates something. The problem
>> was
>> > often that after deprecating something, it gets forgotten.
>> >
>> > For dropping connectors I am a bit uncertain. From a project management
>> > perspective it sounds like a good idea to not have to support connectors
>> > which are no longer supported for some time. However, what if this
>> > connector is still very popular and in heavy use by our users? Just
>> because
>> > an external system or a version of it is no longer maintained does not
>> mean
>> > that the system is no longer used. I think our current practice with
>> trying
>> > to judge whether our users still use this feature/connector works
>> somewhat.
>> > On the other hand, having these guidelines would probably make it
>> easier to
>> > argue to remove something even if there are still a couple of users.
>> >
>> > Cheers,
>> > Till
>> >
>> > On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Very thanks for @Timo to initiate the discussion!
>> > >
>> > > I would also +1 for providing some informations to users via
>> annotations
>> > > or documents in advanced to not suprise users before we actually
>> remove
>> > > the legacy code.
>> > > If we finally decide to change one functionality that user could
>> sense,
>> > > perhaps one
>> > > premise is that Flink has provided a replacement for that one and
>> users
>> > > could transfer their
>> > > applications easily. Then we might also consider have one dedicated
>> > > documentation page
>> > > to list the functionalities to change and how users could do the
>> > transfer.
>> > >
>> > > To make the decision of whether to remove some legacy code, we might
>> also
>> > > consider to have a survey
>> > > like the one we did for mesos support [1] to see how this
>> functionality
>> > is
>> > > used.
>> > >
>> > > Best,
>> > >  Yun
>> > >
>> > >
>> > > [1]
>> > >
>> >
>> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
>> > >
>> > >
>> > >
>> > >  ------------------Original Mail ------------------
>> > > Sender:Piotr Nowojski <[hidden email]>
>> > > Send Date:Mon Jan 18 18:23:36 2021
>> > > Recipients:dev <[hidden email]>
>> > > Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in Flink
>> > > Hi Timo,
>> > >
>> > > Thanks for starting this discussion. I'm not sure how we should
>> approach
>> > > this topic and what should be our final recommendation, but definitely
>> > > clearing up a couple of things would be helpful.
>> > >
>> > > For starters, I agree it would be good to have some more information,
>> > > besides just "@Deprecated" annotations. Is it possible to extend
>> > > annotations with informations like:
>> > > - from which version was it deprecated
>> > > - when is it planned to be removed (we could always mark `2.0` as
>> "never"
>> > > ;) )
>> > > - add maybe some pre/post release step of verifying that removal has
>> > > actually happened?
>> > >
>> > > ?
>> > >
>> > > On the other hand, I think it's very important to maintain backward
>> > > compatibility with Flink as much as possible. As a developer I don't
>> > > like dealing with this, but as a user I hate dealing with incompatible
>> > > upgrades even more. So all in all, I would be in favour of putting
>> more
>> > > effort not in deprecating and removing APIs, but making sure that they
>> > are
>> > > stable.
>> > >
>> > > Stephan Ewan also raised a point sometime ago, that in the recent
>> past,
>> > we
>> > > developed a habit of marking everything as `@Experimental` or
>> > > `@PublicEvolving` and leaving it as that forever. Maybe we should also
>> > > include deadlines (2 releases since introduction?) for changing
>> > > `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
>> > > guidelines/automated checks?
>> > >
>> > > Piotrek
>> > >
>> > > pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]>
>> napisał(a):
>> > >
>> > > > Hi everyone,
>> > > >
>> > > > I would like to start a discussion how we treat deprecated and
>> legacy
>> > > > code in Flink in the future. During the last years, our code base
>> has
>> > > > grown quite a bit and a couple of interfaces and components have
>> been
>> > > > reworked on the way.
>> > > >
>> > > > I'm sure each component has a few legacy parts that are waiting for
>> > > > removal. Apart from keeping outdated API around for a couple of
>> > releases
>> > > > until users have updated their code, it is also often easier to just
>> > put
>> > > > a @Deprecation annotation and postpone the actual change.
>> > > >
>> > > > When looking at the current code, we have duplicated SQL planners,
>> > > > duplicated APIs (DataSet/DataStream), duplicated source/sink
>> > interfaces,
>> > > > outdated connectors (Elasticsearch 5?) and dependencies (Scala
>> 2.11?).
>> > > >
>> > > > I'm wondering whether we should come up with some legacy/deprecation
>> > > > guidelines for the future.
>> > > >
>> > > > Some examples:
>> > > >
>> > > > - I could imagine new Flink-specific annotations for documenting (in
>> > > > code) in which version an interface was deprecated and when the
>> planned
>> > > > removal should take place.
>> > > > - Or guidelines that we drop a connector when the external project
>> does
>> > > > not maintain the version for 6 months etc.
>> > > >
>> > > > Plannable removal dates should also help users to not be surprised
>> when
>> > > > a connector or Scala version is not supported anymore.
>> > > >
>> > > > What do you think? I'm very happy to hear more opinions.
>> > > >
>> > > > Regards,
>> > > > Timo
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>>
>

[Robert Metzger]

Thank you for starting this discussion Timo!

Maybe we should draft a Wiki page outlining our deprecation and removal
policies for legacy code in Flink. We'll need such a document anyways, and
summarizing the consensus from this discussion into a document would allow
us to move forward with this discussion.

Once we agree on a policy, we can do the required code changes (deprecation
interface), update the release manager guidelines etc.

Timo, do you want to drive this?

On Thu, Jan 21, 2021 at 3:37 PM Matthias Pohl <[hidden email]>
wrote:

> >
> > I would prefer not to rely on the Jira for marking when something is
> > supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)`
> would
> > have two obligatory parameters, there would be no way to "forget" about
> > marking it and it would be also self documenting (I don't imagine users
> > using JIRA to check this kind of things). We can have Jira tickets for
> > those things for tracking purposes on the JIRA release board, but relying
> > only on JIRA tickets I think is just asking for inconsistencies.
>
>
> Yes, I totally agree with you. The annotation should be the way to document
> it. I just saw the Jira issue proposal as a good way to track these things
> within the team for code that is not visited regularly. But as I said, that
> depends on how each team organizes itself. I don't see this as a process
> everyone needs to follow.
>
> I agree it would depend on the feature, hence different features might have
> > longer or shorter "unstable" timeframes. But I'm afraid if we won't start
> > thinking about fixing this timeframe, we would too often end up with
> > perpetually "unstable" APIs. I don't know where I would draw the line
> > exactly, but assuming we want to have stable APIs, if something is marked
> > `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be
> switched
> > to `@Public` by default (or be moved out of the main repo?).
>
>
> We could use the timestamp for the @PublicEvolving and @Experimental as an
> indicator to revisit the annotation. Let's say, we revisit each annotation
> after a specific amount of time (e.g. after two releases) to discuss
> whether the API is still evolving or actually stable. If it's stable, we
> change the annotation to @Public. If it's not stable, we update the
> timestamp.
>
> Best,
> Matthias
>
> On Wed, Jan 20, 2021 at 2:01 PM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > I would prefer not to rely on the Jira for marking when something is
> > supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)`
> would
> > have two obligatory parameters, there would be no way to "forget" about
> > marking it and it would be also self documenting (I don't imagine users
> > using JIRA to check this kind of things). We can have Jira tickets for
> > those things for tracking purposes on the JIRA release board, but relying
> > only on JIRA tickets I think is just asking for inconsistencies.
> >
> > > Is it actually possible to have a fixed timeframe for these annotations
> > to change?
> > > I would imagine that it depends on the underlying feature how long an
> > API is @PublicEvolving or @Experimental?
> >
> > I agree it would depend on the feature, hence different features might
> > have longer or shorter "unstable" timeframes. But I'm afraid if we won't
> > start thinking about fixing this timeframe, we would too often end up
> with
> > perpetually "unstable" APIs. I don't know where I would draw the line
> > exactly, but assuming we want to have stable APIs, if something is marked
> > `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be
> switched
> > to `@Public` by default (or be moved out of the main repo?).
> >
> > Piotrek
> >
> > śr., 20 sty 2021 o 09:54 Matthias Pohl <[hidden email]>
> > napisał(a):
> >
> >> Thanks Timo for opening this discussion.
> >>
> >> +1 I like the idea of adding a deprecation deadline and/or information
> >> when
> >> the
> >> functionality was deprecated. It looks like this is already done in the
> >> PyFlink code.
> >>
> >> Creating a JIRA issue for removing the functionality, as Till suggested,
> >> might help to
> >> maintain this process of removing the deprecated functionality. I'd
> prefer
> >> that over
> >> relying on the release manager (assuming that he/she would run the check
> >> as
> >> part
> >> of the release process) to identify functionality that should have been
> >> removed as
> >> part of the release. But ok, that might be a team decision.
> >>
> >> For the connectors: Can't we assume that users would reach out to us if
> we
> >> deprecate
> >> a connector assuming that they can conclude that this connector will,
> >> otherwise, disappear.
> >> Maybe, that needs to be mentioned in the deprecation information as
> well,
> >> then.
> >> This would have the benefit of getting direct feedback about how much a
> >> connector is still in
> >> use and may open the doors for other contributors to offer help like it
> >> happened for the
> >> Mesos support [1].
> >>
> >> And about the idea of adding such deadlines to @Public, @PublicEvolving,
> >> and @Experimental:
> >> Is it actually possible to have a fixed timeframe for these annotations
> to
> >> change? I would
> >> imagine that it depends on the underlying feature how long an API
> >> is @PublicEvolving or
> >> @Experimental? But it sounds still like a good idea to trigger warnings
> >> for
> >> those annotations
> >> in case they haven't been touched for a while. Therefore, I would second
> >> this suggestion.
> >>
> >> Best,
> >> Matthias
> >>
> >> [1]
> >>
> >>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985
> >>
> >> On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <[hidden email]>
> >> wrote:
> >>
> >> > Thanks a lot for starting this discussion Timo. I like the idea of
> >> setting
> >> > more explicit guidelines for deprecating functionality.
> >> >
> >> > I really like the idea of adding with the @Deprecated annotation since
> >> when
> >> > the function is deprecated. Based on that one can simply search for
> >> > features which should be removed in a given release. Alternatively,
> one
> >> > could as you said also state the removal version.
> >> >
> >> > I think what also works is to directly create a critical JIRA issue
> with
> >> > removing functionality as soon as one deprecates something. The
> problem
> >> was
> >> > often that after deprecating something, it gets forgotten.
> >> >
> >> > For dropping connectors I am a bit uncertain. From a project
> management
> >> > perspective it sounds like a good idea to not have to support
> connectors
> >> > which are no longer supported for some time. However, what if this
> >> > connector is still very popular and in heavy use by our users? Just
> >> because
> >> > an external system or a version of it is no longer maintained does not
> >> mean
> >> > that the system is no longer used. I think our current practice with
> >> trying
> >> > to judge whether our users still use this feature/connector works
> >> somewhat.
> >> > On the other hand, having these guidelines would probably make it
> >> easier to
> >> > argue to remove something even if there are still a couple of users.
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]
> >
> >> > wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > Very thanks for @Timo to initiate the discussion!
> >> > >
> >> > > I would also +1 for providing some informations to users via
> >> annotations
> >> > > or documents in advanced to not suprise users before we actually
> >> remove
> >> > > the legacy code.
> >> > > If we finally decide to change one functionality that user could
> >> sense,
> >> > > perhaps one
> >> > > premise is that Flink has provided a replacement for that one and
> >> users
> >> > > could transfer their
> >> > > applications easily. Then we might also consider have one dedicated
> >> > > documentation page
> >> > > to list the functionalities to change and how users could do the
> >> > transfer.
> >> > >
> >> > > To make the decision of whether to remove some legacy code, we might
> >> also
> >> > > consider to have a survey
> >> > > like the one we did for mesos support [1] to see how this
> >> functionality
> >> > is
> >> > > used.
> >> > >
> >> > > Best,
> >> > >  Yun
> >> > >
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
> >> > >
> >> > >
> >> > >
> >> > >  ------------------Original Mail ------------------
> >> > > Sender:Piotr Nowojski <[hidden email]>
> >> > > Send Date:Mon Jan 18 18:23:36 2021
> >> > > Recipients:dev <[hidden email]>
> >> > > Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in
> Flink
> >> > > Hi Timo,
> >> > >
> >> > > Thanks for starting this discussion. I'm not sure how we should
> >> approach
> >> > > this topic and what should be our final recommendation, but
> definitely
> >> > > clearing up a couple of things would be helpful.
> >> > >
> >> > > For starters, I agree it would be good to have some more
> information,
> >> > > besides just "@Deprecated" annotations. Is it possible to extend
> >> > > annotations with informations like:
> >> > > - from which version was it deprecated
> >> > > - when is it planned to be removed (we could always mark `2.0` as
> >> "never"
> >> > > ;) )
> >> > > - add maybe some pre/post release step of verifying that removal has
> >> > > actually happened?
> >> > >
> >> > > ?
> >> > >
> >> > > On the other hand, I think it's very important to maintain backward
> >> > > compatibility with Flink as much as possible. As a developer I don't
> >> > > like dealing with this, but as a user I hate dealing with
> incompatible
> >> > > upgrades even more. So all in all, I would be in favour of putting
> >> more
> >> > > effort not in deprecating and removing APIs, but making sure that
> they
> >> > are
> >> > > stable.
> >> > >
> >> > > Stephan Ewan also raised a point sometime ago, that in the recent
> >> past,
> >> > we
> >> > > developed a habit of marking everything as `@Experimental` or
> >> > > `@PublicEvolving` and leaving it as that forever. Maybe we should
> also
> >> > > include deadlines (2 releases since introduction?) for changing
> >> > > `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
> >> > > guidelines/automated checks?
> >> > >
> >> > > Piotrek
> >> > >
> >> > > pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]>
> >> napisał(a):
> >> > >
> >> > > > Hi everyone,
> >> > > >
> >> > > > I would like to start a discussion how we treat deprecated and
> >> legacy
> >> > > > code in Flink in the future. During the last years, our code base
> >> has
> >> > > > grown quite a bit and a couple of interfaces and components have
> >> been
> >> > > > reworked on the way.
> >> > > >
> >> > > > I'm sure each component has a few legacy parts that are waiting
> for
> >> > > > removal. Apart from keeping outdated API around for a couple of
> >> > releases
> >> > > > until users have updated their code, it is also often easier to
> just
> >> > put
> >> > > > a @Deprecation annotation and postpone the actual change.
> >> > > >
> >> > > > When looking at the current code, we have duplicated SQL planners,
> >> > > > duplicated APIs (DataSet/DataStream), duplicated source/sink
> >> > interfaces,
> >> > > > outdated connectors (Elasticsearch 5?) and dependencies (Scala
> >> 2.11?).
> >> > > >
> >> > > > I'm wondering whether we should come up with some
> legacy/deprecation
> >> > > > guidelines for the future.
> >> > > >
> >> > > > Some examples:
> >> > > >
> >> > > > - I could imagine new Flink-specific annotations for documenting
> (in
> >> > > > code) in which version an interface was deprecated and when the
> >> planned
> >> > > > removal should take place.
> >> > > > - Or guidelines that we drop a connector when the external project
> >> does
> >> > > > not maintain the version for 6 months etc.
> >> > > >
> >> > > > Plannable removal dates should also help users to not be surprised
> >> when
> >> > > > a connector or Scala version is not supported anymore.
> >> > > >
> >> > > > What do you think? I'm very happy to hear more opinions.
> >> > > >
> >> > > > Regards,
> >> > > > Timo
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >>
> >
>

[Timo Walther-2]

Hi everyone,

great to see all these responses. I'm happy to drive this effort. I will
try to summarize this discussion and look into how other projects are
handling this topic. A might need a couple days more for this. After
that we can discuss how we would like our guidlines to look like.

Thanks,
Timo

On 26.01.21 14:28, Robert Metzger wrote:

> Thank you for starting this discussion Timo!
>
> Maybe we should draft a Wiki page outlining our deprecation and removal
> policies for legacy code in Flink. We'll need such a document anyways, and
> summarizing the consensus from this discussion into a document would allow
> us to move forward with this discussion.
>
> Once we agree on a policy, we can do the required code changes (deprecation
> interface), update the release manager guidelines etc.
>
> Timo, do you want to drive this?
>
> On Thu, Jan 21, 2021 at 3:37 PM Matthias Pohl <[hidden email]>
> wrote:
>
>>>
>>> I would prefer not to rely on the Jira for marking when something is
>>> supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)`
>> would
>>> have two obligatory parameters, there would be no way to "forget" about
>>> marking it and it would be also self documenting (I don't imagine users
>>> using JIRA to check this kind of things). We can have Jira tickets for
>>> those things for tracking purposes on the JIRA release board, but relying
>>> only on JIRA tickets I think is just asking for inconsistencies.
>>
>>
>> Yes, I totally agree with you. The annotation should be the way to document
>> it. I just saw the Jira issue proposal as a good way to track these things
>> within the team for code that is not visited regularly. But as I said, that
>> depends on how each team organizes itself. I don't see this as a process
>> everyone needs to follow.
>>
>> I agree it would depend on the feature, hence different features might have
>>> longer or shorter "unstable" timeframes. But I'm afraid if we won't start
>>> thinking about fixing this timeframe, we would too often end up with
>>> perpetually "unstable" APIs. I don't know where I would draw the line
>>> exactly, but assuming we want to have stable APIs, if something is marked
>>> `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be
>> switched
>>> to `@Public` by default (or be moved out of the main repo?).
>>
>>
>> We could use the timestamp for the @PublicEvolving and @Experimental as an
>> indicator to revisit the annotation. Let's say, we revisit each annotation
>> after a specific amount of time (e.g. after two releases) to discuss
>> whether the API is still evolving or actually stable. If it's stable, we
>> change the annotation to @Public. If it's not stable, we update the
>> timestamp.
>>
>> Best,
>> Matthias
>>
>> On Wed, Jan 20, 2021 at 2:01 PM Piotr Nowojski <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> I would prefer not to rely on the Jira for marking when something is
>>> supposed to be deleted. If `@Deprecated(since, planned_to_remove_on)`
>> would
>>> have two obligatory parameters, there would be no way to "forget" about
>>> marking it and it would be also self documenting (I don't imagine users
>>> using JIRA to check this kind of things). We can have Jira tickets for
>>> those things for tracking purposes on the JIRA release board, but relying
>>> only on JIRA tickets I think is just asking for inconsistencies.
>>>
>>>> Is it actually possible to have a fixed timeframe for these annotations
>>> to change?
>>>> I would imagine that it depends on the underlying feature how long an
>>> API is @PublicEvolving or @Experimental?
>>>
>>> I agree it would depend on the feature, hence different features might
>>> have longer or shorter "unstable" timeframes. But I'm afraid if we won't
>>> start thinking about fixing this timeframe, we would too often end up
>> with
>>> perpetually "unstable" APIs. I don't know where I would draw the line
>>> exactly, but assuming we want to have stable APIs, if something is marked
>>> `@PublicEvolving` or `@Experimental` for 3 years, IMO it should be
>> switched
>>> to `@Public` by default (or be moved out of the main repo?).
>>>
>>> Piotrek
>>>
>>> śr., 20 sty 2021 o 09:54 Matthias Pohl <[hidden email]>
>>> napisał(a):
>>>
>>>> Thanks Timo for opening this discussion.
>>>>
>>>> +1 I like the idea of adding a deprecation deadline and/or information
>>>> when
>>>> the
>>>> functionality was deprecated. It looks like this is already done in the
>>>> PyFlink code.
>>>>
>>>> Creating a JIRA issue for removing the functionality, as Till suggested,
>>>> might help to
>>>> maintain this process of removing the deprecated functionality. I'd
>> prefer
>>>> that over
>>>> relying on the release manager (assuming that he/she would run the check
>>>> as
>>>> part
>>>> of the release process) to identify functionality that should have been
>>>> removed as
>>>> part of the release. But ok, that might be a team decision.
>>>>
>>>> For the connectors: Can't we assume that users would reach out to us if
>> we
>>>> deprecate
>>>> a connector assuming that they can conclude that this connector will,
>>>> otherwise, disappear.
>>>> Maybe, that needs to be mentioned in the deprecation information as
>> well,
>>>> then.
>>>> This would have the benefit of getting direct feedback about how much a
>>>> connector is still in
>>>> use and may open the doors for other contributors to offer help like it
>>>> happened for the
>>>> Mesos support [1].
>>>>
>>>> And about the idea of adding such deadlines to @Public, @PublicEvolving,
>>>> and @Experimental:
>>>> Is it actually possible to have a fixed timeframe for these annotations
>> to
>>>> change? I would
>>>> imagine that it depends on the underlying feature how long an API
>>>> is @PublicEvolving or
>>>> @Experimental? But it sounds still like a good idea to trigger warnings
>>>> for
>>>> those annotations
>>>> in case they haven't been touched for a while. Therefore, I would second
>>>> this suggestion.
>>>>
>>>> Best,
>>>> Matthias
>>>>
>>>> [1]
>>>>
>>>>
>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html#a45985
>>>>
>>>> On Tue, Jan 19, 2021 at 10:15 AM Till Rohrmann <[hidden email]>
>>>> wrote:
>>>>
>>>>> Thanks a lot for starting this discussion Timo. I like the idea of
>>>> setting
>>>>> more explicit guidelines for deprecating functionality.
>>>>>
>>>>> I really like the idea of adding with the @Deprecated annotation since
>>>> when
>>>>> the function is deprecated. Based on that one can simply search for
>>>>> features which should be removed in a given release. Alternatively,
>> one
>>>>> could as you said also state the removal version.
>>>>>
>>>>> I think what also works is to directly create a critical JIRA issue
>> with
>>>>> removing functionality as soon as one deprecates something. The
>> problem
>>>> was
>>>>> often that after deprecating something, it gets forgotten.
>>>>>
>>>>> For dropping connectors I am a bit uncertain. From a project
>> management
>>>>> perspective it sounds like a good idea to not have to support
>> connectors
>>>>> which are no longer supported for some time. However, what if this
>>>>> connector is still very popular and in heavy use by our users? Just
>>>> because
>>>>> an external system or a version of it is no longer maintained does not
>>>> mean
>>>>> that the system is no longer used. I think our current practice with
>>>> trying
>>>>> to judge whether our users still use this feature/connector works
>>>> somewhat.
>>>>> On the other hand, having these guidelines would probably make it
>>>> easier to
>>>>> argue to remove something even if there are still a couple of users.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Mon, Jan 18, 2021 at 11:37 AM Yun Gao <[hidden email]
>>>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Very thanks for @Timo to initiate the discussion!
>>>>>>
>>>>>> I would also +1 for providing some informations to users via
>>>> annotations
>>>>>> or documents in advanced to not suprise users before we actually
>>>> remove
>>>>>> the legacy code.
>>>>>> If we finally decide to change one functionality that user could
>>>> sense,
>>>>>> perhaps one
>>>>>> premise is that Flink has provided a replacement for that one and
>>>> users
>>>>>> could transfer their
>>>>>> applications easily. Then we might also consider have one dedicated
>>>>>> documentation page
>>>>>> to list the functionalities to change and how users could do the
>>>>> transfer.
>>>>>>
>>>>>> To make the decision of whether to remove some legacy code, we might
>>>> also
>>>>>> consider to have a survey
>>>>>> like the one we did for mesos support [1] to see how this
>>>> functionality
>>>>> is
>>>>>> used.
>>>>>>
>>>>>> Best,
>>>>>>   Yun
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>
>>>>
>> https://lists.apache.org/thread.html/r139b11190a6d1f09c9e44d5fa985fd8d310347e66d2324ec1f0c2d87%40%3Cuser.flink.apache.org%3E
>>>>>>
>>>>>>
>>>>>>
>>>>>>   ------------------Original Mail ------------------
>>>>>> Sender:Piotr Nowojski <[hidden email]>
>>>>>> Send Date:Mon Jan 18 18:23:36 2021
>>>>>> Recipients:dev <[hidden email]>
>>>>>> Subject:Re: [DISCUSS] Dealing with deprecated and legacy code in
>> Flink
>>>>>> Hi Timo,
>>>>>>
>>>>>> Thanks for starting this discussion. I'm not sure how we should
>>>> approach
>>>>>> this topic and what should be our final recommendation, but
>> definitely
>>>>>> clearing up a couple of things would be helpful.
>>>>>>
>>>>>> For starters, I agree it would be good to have some more
>> information,
>>>>>> besides just "@Deprecated" annotations. Is it possible to extend
>>>>>> annotations with informations like:
>>>>>> - from which version was it deprecated
>>>>>> - when is it planned to be removed (we could always mark `2.0` as
>>>> "never"
>>>>>> ;) )
>>>>>> - add maybe some pre/post release step of verifying that removal has
>>>>>> actually happened?
>>>>>>
>>>>>> ?
>>>>>>
>>>>>> On the other hand, I think it's very important to maintain backward
>>>>>> compatibility with Flink as much as possible. As a developer I don't
>>>>>> like dealing with this, but as a user I hate dealing with
>> incompatible
>>>>>> upgrades even more. So all in all, I would be in favour of putting
>>>> more
>>>>>> effort not in deprecating and removing APIs, but making sure that
>> they
>>>>> are
>>>>>> stable.
>>>>>>
>>>>>> Stephan Ewan also raised a point sometime ago, that in the recent
>>>> past,
>>>>> we
>>>>>> developed a habit of marking everything as `@Experimental` or
>>>>>> `@PublicEvolving` and leaving it as that forever. Maybe we should
>> also
>>>>>> include deadlines (2 releases since introduction?) for changing
>>>>>> `@Experimental`/`@PublicEvolving` into `@Public` in this kind of
>>>>>> guidelines/automated checks?
>>>>>>
>>>>>> Piotrek
>>>>>>
>>>>>> pt., 15 sty 2021 o 13:56 Timo Walther <[hidden email]>
>>>> napisał(a):
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I would like to start a discussion how we treat deprecated and
>>>> legacy
>>>>>>> code in Flink in the future. During the last years, our code base
>>>> has
>>>>>>> grown quite a bit and a couple of interfaces and components have
>>>> been
>>>>>>> reworked on the way.
>>>>>>>
>>>>>>> I'm sure each component has a few legacy parts that are waiting
>> for
>>>>>>> removal. Apart from keeping outdated API around for a couple of
>>>>> releases
>>>>>>> until users have updated their code, it is also often easier to
>> just
>>>>> put
>>>>>>> a @Deprecation annotation and postpone the actual change.
>>>>>>>
>>>>>>> When looking at the current code, we have duplicated SQL planners,
>>>>>>> duplicated APIs (DataSet/DataStream), duplicated source/sink
>>>>> interfaces,
>>>>>>> outdated connectors (Elasticsearch 5?) and dependencies (Scala
>>>> 2.11?).
>>>>>>>
>>>>>>> I'm wondering whether we should come up with some
>> legacy/deprecation
>>>>>>> guidelines for the future.
>>>>>>>
>>>>>>> Some examples:
>>>>>>>
>>>>>>> - I could imagine new Flink-specific annotations for documenting
>> (in
>>>>>>> code) in which version an interface was deprecated and when the
>>>> planned
>>>>>>> removal should take place.
>>>>>>> - Or guidelines that we drop a connector when the external project
>>>> does
>>>>>>> not maintain the version for 6 months etc.
>>>>>>>
>>>>>>> Plannable removal dates should also help users to not be surprised
>>>> when
>>>>>>> a connector or Scala version is not supported anymore.
>>>>>>>
>>>>>>> What do you think? I'm very happy to hear more opinions.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Timo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>