[FLINK-305] Code test coverage - how FLINK using it?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[FLINK-305] Code test coverage - how FLINK using it?

Pavel Fadeev
Dear team,

I`m just looking around into the project - complete novice at Flink :)
Sorry if Qs below already have answers!

At first glance I`ve discovered that code coverage feature has been
introduced with FLINK-305 <https://issues.apache.org/jira/browse/FLINK-305>
and then removed for some reason on March 2015.

Are you aware if it is not required anymore? Worried a bit about this after
local coverage run for flink-code as well. Do we have some integration like
codecov <https://codecov.io> here and do you feel if it is required?

Also, do you know if there are some statistics (or team knowledge) for
regression bugs from uncovered code?
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

Fabian Hueske-2
Hi Pavel,

the Cobertura plugin was removed in this PR:
https://github.com/apache/flink/pull/454
Not sure if it was accidentally removed or on purpose.
It was not included in the regular builds to reduce build time and AFAIK,
it wasn't manually used either (otherwise somebody would have noticed that
it is gone).

I am not aware of any code coverage stats for the Flink code base but it
would be nice to some, IMO.

Best, Fabian

2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:

> Dear team,
>
> I`m just looking around into the project - complete novice at Flink :)
> Sorry if Qs below already have answers!
>
> At first glance I`ve discovered that code coverage feature has been
> introduced with FLINK-305 <https://issues.apache.org/jira/browse/FLINK-305
> >
> and then removed for some reason on March 2015.
>
> Are you aware if it is not required anymore? Worried a bit about this after
> local coverage run for flink-code as well. Do we have some integration like
> codecov <https://codecov.io> here and do you feel if it is required?
>
> Also, do you know if there are some statistics (or team knowledge) for
> regression bugs from uncovered code?
>
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

Till Rohrmann
Hi Pavel,

I think it's a good point you're raising here. The Flink community isn't
using metrics like test coverage to ensure high quality code yet. I think
that is one thing which we can/should improve. Unfortunately, the ASF does
not allow to use codecov.io (or at least the Apache Infra team) [1].
However, they encourage to use coveralls.io.

Do you know this tool and want to take a look how it could be integrated
with the Flink repository? It seems as if it is free for open source
projects. Maybe you can create a JIRA issue for this integration and then
take the lead there.

[1] https://issues.apache.org/jira/browse/INFRA-11273

Cheers,
Till

On Fri, Aug 19, 2016 at 11:33 AM, Fabian Hueske <[hidden email]> wrote:

> Hi Pavel,
>
> the Cobertura plugin was removed in this PR:
> https://github.com/apache/flink/pull/454
> Not sure if it was accidentally removed or on purpose.
> It was not included in the regular builds to reduce build time and AFAIK,
> it wasn't manually used either (otherwise somebody would have noticed that
> it is gone).
>
> I am not aware of any code coverage stats for the Flink code base but it
> would be nice to some, IMO.
>
> Best, Fabian
>
> 2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:
>
> > Dear team,
> >
> > I`m just looking around into the project - complete novice at Flink :)
> > Sorry if Qs below already have answers!
> >
> > At first glance I`ve discovered that code coverage feature has been
> > introduced with FLINK-305 <https://issues.apache.org/
> jira/browse/FLINK-305
> > >
> > and then removed for some reason on March 2015.
> >
> > Are you aware if it is not required anymore? Worried a bit about this
> after
> > local coverage run for flink-code as well. Do we have some integration
> like
> > codecov <https://codecov.io> here and do you feel if it is required?
> >
> > Also, do you know if there are some statistics (or team knowledge) for
> > regression bugs from uncovered code?
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

Pavel Fadeev
Fabian, Till,
thanks for your comments!

I`ve raised https://issues.apache.org/jira/browse/INFRA-12458 for the same.
Will play around to see if this solution acceptable and does not affect
build duration too much.

2016-08-19 12:51 GMT+03:00 Till Rohrmann <[hidden email]>:

> Hi Pavel,
>
> I think it's a good point you're raising here. The Flink community isn't
> using metrics like test coverage to ensure high quality code yet. I think
> that is one thing which we can/should improve. Unfortunately, the ASF does
> not allow to use codecov.io (or at least the Apache Infra team) [1].
> However, they encourage to use coveralls.io.
>
> Do you know this tool and want to take a look how it could be integrated
> with the Flink repository? It seems as if it is free for open source
> projects. Maybe you can create a JIRA issue for this integration and then
> take the lead there.
>
> [1] https://issues.apache.org/jira/browse/INFRA-11273
>
> Cheers,
> Till
>
> On Fri, Aug 19, 2016 at 11:33 AM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi Pavel,
> >
> > the Cobertura plugin was removed in this PR:
> > https://github.com/apache/flink/pull/454
> > Not sure if it was accidentally removed or on purpose.
> > It was not included in the regular builds to reduce build time and AFAIK,
> > it wasn't manually used either (otherwise somebody would have noticed
> that
> > it is gone).
> >
> > I am not aware of any code coverage stats for the Flink code base but it
> > would be nice to some, IMO.
> >
> > Best, Fabian
> >
> > 2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:
> >
> > > Dear team,
> > >
> > > I`m just looking around into the project - complete novice at Flink :)
> > > Sorry if Qs below already have answers!
> > >
> > > At first glance I`ve discovered that code coverage feature has been
> > > introduced with FLINK-305 <https://issues.apache.org/
> > jira/browse/FLINK-305
> > > >
> > > and then removed for some reason on March 2015.
> > >
> > > Are you aware if it is not required anymore? Worried a bit about this
> > after
> > > local coverage run for flink-code as well. Do we have some integration
> > like
> > > codecov <https://codecov.io> here and do you feel if it is required?
> > >
> > > Also, do you know if there are some statistics (or team knowledge) for
> > > regression bugs from uncovered code?
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

Till Rohrmann
Thanks a lot for your help with that Pavel :-)

On Fri, Aug 19, 2016 at 9:48 PM, Pavel Fadeev <[hidden email]>
wrote:

> Fabian, Till,
> thanks for your comments!
>
> I`ve raised https://issues.apache.org/jira/browse/INFRA-12458 for the
> same.
> Will play around to see if this solution acceptable and does not affect
> build duration too much.
>
> 2016-08-19 12:51 GMT+03:00 Till Rohrmann <[hidden email]>:
>
> > Hi Pavel,
> >
> > I think it's a good point you're raising here. The Flink community isn't
> > using metrics like test coverage to ensure high quality code yet. I think
> > that is one thing which we can/should improve. Unfortunately, the ASF
> does
> > not allow to use codecov.io (or at least the Apache Infra team) [1].
> > However, they encourage to use coveralls.io.
> >
> > Do you know this tool and want to take a look how it could be integrated
> > with the Flink repository? It seems as if it is free for open source
> > projects. Maybe you can create a JIRA issue for this integration and then
> > take the lead there.
> >
> > [1] https://issues.apache.org/jira/browse/INFRA-11273
> >
> > Cheers,
> > Till
> >
> > On Fri, Aug 19, 2016 at 11:33 AM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi Pavel,
> > >
> > > the Cobertura plugin was removed in this PR:
> > > https://github.com/apache/flink/pull/454
> > > Not sure if it was accidentally removed or on purpose.
> > > It was not included in the regular builds to reduce build time and
> AFAIK,
> > > it wasn't manually used either (otherwise somebody would have noticed
> > that
> > > it is gone).
> > >
> > > I am not aware of any code coverage stats for the Flink code base but
> it
> > > would be nice to some, IMO.
> > >
> > > Best, Fabian
> > >
> > > 2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:
> > >
> > > > Dear team,
> > > >
> > > > I`m just looking around into the project - complete novice at Flink
> :)
> > > > Sorry if Qs below already have answers!
> > > >
> > > > At first glance I`ve discovered that code coverage feature has been
> > > > introduced with FLINK-305 <https://issues.apache.org/
> > > jira/browse/FLINK-305
> > > > >
> > > > and then removed for some reason on March 2015.
> > > >
> > > > Are you aware if it is not required anymore? Worried a bit about this
> > > after
> > > > local coverage run for flink-code as well. Do we have some
> integration
> > > like
> > > > codecov <https://codecov.io> here and do you feel if it is required?
> > > >
> > > > Also, do you know if there are some statistics (or team knowledge)
> for
> > > > regression bugs from uncovered code?
> > > >
> > >
> >
>
mxm
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

mxm
Hi Pavel!

Thanks for looking into code coverage! Now that Infra enabled access
to coveralls, could you open a Flink issue to address the next steps
to display coverage data?

Cheers,
Max

On Mon, Aug 22, 2016 at 11:38 AM, Till Rohrmann <[hidden email]> wrote:

> Thanks a lot for your help with that Pavel :-)
>
> On Fri, Aug 19, 2016 at 9:48 PM, Pavel Fadeev <[hidden email]>
> wrote:
>
>> Fabian, Till,
>> thanks for your comments!
>>
>> I`ve raised https://issues.apache.org/jira/browse/INFRA-12458 for the
>> same.
>> Will play around to see if this solution acceptable and does not affect
>> build duration too much.
>>
>> 2016-08-19 12:51 GMT+03:00 Till Rohrmann <[hidden email]>:
>>
>> > Hi Pavel,
>> >
>> > I think it's a good point you're raising here. The Flink community isn't
>> > using metrics like test coverage to ensure high quality code yet. I think
>> > that is one thing which we can/should improve. Unfortunately, the ASF
>> does
>> > not allow to use codecov.io (or at least the Apache Infra team) [1].
>> > However, they encourage to use coveralls.io.
>> >
>> > Do you know this tool and want to take a look how it could be integrated
>> > with the Flink repository? It seems as if it is free for open source
>> > projects. Maybe you can create a JIRA issue for this integration and then
>> > take the lead there.
>> >
>> > [1] https://issues.apache.org/jira/browse/INFRA-11273
>> >
>> > Cheers,
>> > Till
>> >
>> > On Fri, Aug 19, 2016 at 11:33 AM, Fabian Hueske <[hidden email]>
>> wrote:
>> >
>> > > Hi Pavel,
>> > >
>> > > the Cobertura plugin was removed in this PR:
>> > > https://github.com/apache/flink/pull/454
>> > > Not sure if it was accidentally removed or on purpose.
>> > > It was not included in the regular builds to reduce build time and
>> AFAIK,
>> > > it wasn't manually used either (otherwise somebody would have noticed
>> > that
>> > > it is gone).
>> > >
>> > > I am not aware of any code coverage stats for the Flink code base but
>> it
>> > > would be nice to some, IMO.
>> > >
>> > > Best, Fabian
>> > >
>> > > 2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:
>> > >
>> > > > Dear team,
>> > > >
>> > > > I`m just looking around into the project - complete novice at Flink
>> :)
>> > > > Sorry if Qs below already have answers!
>> > > >
>> > > > At first glance I`ve discovered that code coverage feature has been
>> > > > introduced with FLINK-305 <https://issues.apache.org/
>> > > jira/browse/FLINK-305
>> > > > >
>> > > > and then removed for some reason on March 2015.
>> > > >
>> > > > Are you aware if it is not required anymore? Worried a bit about this
>> > > after
>> > > > local coverage run for flink-code as well. Do we have some
>> integration
>> > > like
>> > > > codecov <https://codecov.io> here and do you feel if it is required?
>> > > >
>> > > > Also, do you know if there are some statistics (or team knowledge)
>> for
>> > > > regression bugs from uncovered code?
>> > > >
>> > >
>> >
>>
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

Pavel Fadeev
Hi Maximilian,
absolutely make sense! Just filed
https://issues.apache.org/jira/browse/FLINK-4483 for the same.

More than happy to go ahead as an assignee there, but not authorized for
this kind of action atm)

2016-08-24 13:32 GMT+03:00 Maximilian Michels <[hidden email]>:

> Hi Pavel!
>
> Thanks for looking into code coverage! Now that Infra enabled access
> to coveralls, could you open a Flink issue to address the next steps
> to display coverage data?
>
> Cheers,
> Max
>
> On Mon, Aug 22, 2016 at 11:38 AM, Till Rohrmann <[hidden email]>
> wrote:
> > Thanks a lot for your help with that Pavel :-)
> >
> > On Fri, Aug 19, 2016 at 9:48 PM, Pavel Fadeev <[hidden email]>
> > wrote:
> >
> >> Fabian, Till,
> >> thanks for your comments!
> >>
> >> I`ve raised https://issues.apache.org/jira/browse/INFRA-12458 for the
> >> same.
> >> Will play around to see if this solution acceptable and does not affect
> >> build duration too much.
> >>
> >> 2016-08-19 12:51 GMT+03:00 Till Rohrmann <[hidden email]>:
> >>
> >> > Hi Pavel,
> >> >
> >> > I think it's a good point you're raising here. The Flink community
> isn't
> >> > using metrics like test coverage to ensure high quality code yet. I
> think
> >> > that is one thing which we can/should improve. Unfortunately, the ASF
> >> does
> >> > not allow to use codecov.io (or at least the Apache Infra team) [1].
> >> > However, they encourage to use coveralls.io.
> >> >
> >> > Do you know this tool and want to take a look how it could be
> integrated
> >> > with the Flink repository? It seems as if it is free for open source
> >> > projects. Maybe you can create a JIRA issue for this integration and
> then
> >> > take the lead there.
> >> >
> >> > [1] https://issues.apache.org/jira/browse/INFRA-11273
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Fri, Aug 19, 2016 at 11:33 AM, Fabian Hueske <[hidden email]>
> >> wrote:
> >> >
> >> > > Hi Pavel,
> >> > >
> >> > > the Cobertura plugin was removed in this PR:
> >> > > https://github.com/apache/flink/pull/454
> >> > > Not sure if it was accidentally removed or on purpose.
> >> > > It was not included in the regular builds to reduce build time and
> >> AFAIK,
> >> > > it wasn't manually used either (otherwise somebody would have
> noticed
> >> > that
> >> > > it is gone).
> >> > >
> >> > > I am not aware of any code coverage stats for the Flink code base
> but
> >> it
> >> > > would be nice to some, IMO.
> >> > >
> >> > > Best, Fabian
> >> > >
> >> > > 2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:
> >> > >
> >> > > > Dear team,
> >> > > >
> >> > > > I`m just looking around into the project - complete novice at
> Flink
> >> :)
> >> > > > Sorry if Qs below already have answers!
> >> > > >
> >> > > > At first glance I`ve discovered that code coverage feature has
> been
> >> > > > introduced with FLINK-305 <https://issues.apache.org/
> >> > > jira/browse/FLINK-305
> >> > > > >
> >> > > > and then removed for some reason on March 2015.
> >> > > >
> >> > > > Are you aware if it is not required anymore? Worried a bit about
> this
> >> > > after
> >> > > > local coverage run for flink-code as well. Do we have some
> >> integration
> >> > > like
> >> > > > codecov <https://codecov.io> here and do you feel if it is
> required?
> >> > > >
> >> > > > Also, do you know if there are some statistics (or team knowledge)
> >> for
> >> > > > regression bugs from uncovered code?
> >> > > >
> >> > >
> >> >
> >>
>
mxm
Reply | Threaded
Open this post in threaded view
|

Re: [FLINK-305] Code test coverage - how FLINK using it?

mxm
That should be solved now. Looking forward to the pull request :)

On Thu, Aug 25, 2016 at 1:26 AM, Pavel Fadeev <[hidden email]> wrote:

> Hi Maximilian,
> absolutely make sense! Just filed
> https://issues.apache.org/jira/browse/FLINK-4483 for the same.
>
> More than happy to go ahead as an assignee there, but not authorized for
> this kind of action atm)
>
> 2016-08-24 13:32 GMT+03:00 Maximilian Michels <[hidden email]>:
>
>> Hi Pavel!
>>
>> Thanks for looking into code coverage! Now that Infra enabled access
>> to coveralls, could you open a Flink issue to address the next steps
>> to display coverage data?
>>
>> Cheers,
>> Max
>>
>> On Mon, Aug 22, 2016 at 11:38 AM, Till Rohrmann <[hidden email]>
>> wrote:
>> > Thanks a lot for your help with that Pavel :-)
>> >
>> > On Fri, Aug 19, 2016 at 9:48 PM, Pavel Fadeev <[hidden email]>
>> > wrote:
>> >
>> >> Fabian, Till,
>> >> thanks for your comments!
>> >>
>> >> I`ve raised https://issues.apache.org/jira/browse/INFRA-12458 for the
>> >> same.
>> >> Will play around to see if this solution acceptable and does not affect
>> >> build duration too much.
>> >>
>> >> 2016-08-19 12:51 GMT+03:00 Till Rohrmann <[hidden email]>:
>> >>
>> >> > Hi Pavel,
>> >> >
>> >> > I think it's a good point you're raising here. The Flink community
>> isn't
>> >> > using metrics like test coverage to ensure high quality code yet. I
>> think
>> >> > that is one thing which we can/should improve. Unfortunately, the ASF
>> >> does
>> >> > not allow to use codecov.io (or at least the Apache Infra team) [1].
>> >> > However, they encourage to use coveralls.io.
>> >> >
>> >> > Do you know this tool and want to take a look how it could be
>> integrated
>> >> > with the Flink repository? It seems as if it is free for open source
>> >> > projects. Maybe you can create a JIRA issue for this integration and
>> then
>> >> > take the lead there.
>> >> >
>> >> > [1] https://issues.apache.org/jira/browse/INFRA-11273
>> >> >
>> >> > Cheers,
>> >> > Till
>> >> >
>> >> > On Fri, Aug 19, 2016 at 11:33 AM, Fabian Hueske <[hidden email]>
>> >> wrote:
>> >> >
>> >> > > Hi Pavel,
>> >> > >
>> >> > > the Cobertura plugin was removed in this PR:
>> >> > > https://github.com/apache/flink/pull/454
>> >> > > Not sure if it was accidentally removed or on purpose.
>> >> > > It was not included in the regular builds to reduce build time and
>> >> AFAIK,
>> >> > > it wasn't manually used either (otherwise somebody would have
>> noticed
>> >> > that
>> >> > > it is gone).
>> >> > >
>> >> > > I am not aware of any code coverage stats for the Flink code base
>> but
>> >> it
>> >> > > would be nice to some, IMO.
>> >> > >
>> >> > > Best, Fabian
>> >> > >
>> >> > > 2016-08-19 0:36 GMT+02:00 Pavel Fadeev <[hidden email]>:
>> >> > >
>> >> > > > Dear team,
>> >> > > >
>> >> > > > I`m just looking around into the project - complete novice at
>> Flink
>> >> :)
>> >> > > > Sorry if Qs below already have answers!
>> >> > > >
>> >> > > > At first glance I`ve discovered that code coverage feature has
>> been
>> >> > > > introduced with FLINK-305 <https://issues.apache.org/
>> >> > > jira/browse/FLINK-305
>> >> > > > >
>> >> > > > and then removed for some reason on March 2015.
>> >> > > >
>> >> > > > Are you aware if it is not required anymore? Worried a bit about
>> this
>> >> > > after
>> >> > > > local coverage run for flink-code as well. Do we have some
>> >> integration
>> >> > > like
>> >> > > > codecov <https://codecov.io> here and do you feel if it is
>> required?
>> >> > > >
>> >> > > > Also, do you know if there are some statistics (or team knowledge)
>> >> for
>> >> > > > regression bugs from uncovered code?
>> >> > > >
>> >> > >
>> >> >
>> >>
>>