[DISCUSS] Collecting Instable Tests

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Collecting Instable Tests

Matthias J. Sax-2
Hi,

right now we collect instable tests via labeling them with "test-stability".

I think it would be a better approach, to use an Umbrella Issue in JIRA
and collect them as Sub-Tasks. Storm does the same thing:
https://issues.apache.org/jira/browse/STORM-915

This would have the advantage, that we get a simpler overview what tests
got already reported (and/or fixed). We can also list the tests by
component which makes it easier to find (the JIRA "search issues"
feature is cumbersome to use because the UI is quite slow)

I would keep current approach and label the sub-tasks with
"test-stability" and mark them as "critical".

What do you think about it?


-Matthias


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Collecting Instable Tests

Fabian Hueske-2
What would be the benefit over the current approach?
You can just fiter JIRA for issues with a specific label:

https://issues.apache.org/jira/browse/FLINK-2771?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC

Not sure if the gain (if there is any) justifies the overhead of adding
issues to an umbrella issue.


2015-10-09 13:47 GMT+02:00 Matthias J. Sax <[hidden email]>:

> Hi,
>
> right now we collect instable tests via labeling them with
> "test-stability".
>
> I think it would be a better approach, to use an Umbrella Issue in JIRA
> and collect them as Sub-Tasks. Storm does the same thing:
> https://issues.apache.org/jira/browse/STORM-915
>
> This would have the advantage, that we get a simpler overview what tests
> got already reported (and/or fixed). We can also list the tests by
> component which makes it easier to find (the JIRA "search issues"
> feature is cumbersome to use because the UI is quite slow)
>
> I would keep current approach and label the sub-tasks with
> "test-stability" and mark them as "critical".
>
> What do you think about it?
>
>
> -Matthias
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Collecting Instable Tests

Matthias J. Sax-2
Yes, but this filtering is super slow (or is it just my browser?)

I personally don't like the search feature in JIRA and an umbrella would
give a easier to access overview.

But if it's just me, never mind ;)


On 10/09/2015 02:39 PM, Fabian Hueske wrote:

> What would be the benefit over the current approach?
> You can just fiter JIRA for issues with a specific label:
>
> https://issues.apache.org/jira/browse/FLINK-2771?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC
>
> Not sure if the gain (if there is any) justifies the overhead of adding
> issues to an umbrella issue.
>
>
> 2015-10-09 13:47 GMT+02:00 Matthias J. Sax <[hidden email]>:
>
>> Hi,
>>
>> right now we collect instable tests via labeling them with
>> "test-stability".
>>
>> I think it would be a better approach, to use an Umbrella Issue in JIRA
>> and collect them as Sub-Tasks. Storm does the same thing:
>> https://issues.apache.org/jira/browse/STORM-915
>>
>> This would have the advantage, that we get a simpler overview what tests
>> got already reported (and/or fixed). We can also list the tests by
>> component which makes it easier to find (the JIRA "search issues"
>> feature is cumbersome to use because the UI is quite slow)
>>
>> I would keep current approach and label the sub-tasks with
>> "test-stability" and mark them as "critical".
>>
>> What do you think about it?
>>
>>
>> -Matthias
>>
>>
>


signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Collecting Instable Tests

Fabian Hueske-2
Sorry, that was not the correct link.
You can create a bookmark for this one:

https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC

and directly ask for the list without searching.

2015-10-09 14:48 GMT+02:00 Matthias J. Sax <[hidden email]>:

> Yes, but this filtering is super slow (or is it just my browser?)
>
> I personally don't like the search feature in JIRA and an umbrella would
> give a easier to access overview.
>
> But if it's just me, never mind ;)
>
>
> On 10/09/2015 02:39 PM, Fabian Hueske wrote:
> > What would be the benefit over the current approach?
> > You can just fiter JIRA for issues with a specific label:
> >
> >
> https://issues.apache.org/jira/browse/FLINK-2771?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC
> >
> > Not sure if the gain (if there is any) justifies the overhead of adding
> > issues to an umbrella issue.
> >
> >
> > 2015-10-09 13:47 GMT+02:00 Matthias J. Sax <[hidden email]>:
> >
> >> Hi,
> >>
> >> right now we collect instable tests via labeling them with
> >> "test-stability".
> >>
> >> I think it would be a better approach, to use an Umbrella Issue in JIRA
> >> and collect them as Sub-Tasks. Storm does the same thing:
> >> https://issues.apache.org/jira/browse/STORM-915
> >>
> >> This would have the advantage, that we get a simpler overview what tests
> >> got already reported (and/or fixed). We can also list the tests by
> >> component which makes it easier to find (the JIRA "search issues"
> >> feature is cumbersome to use because the UI is quite slow)
> >>
> >> I would keep current approach and label the sub-tasks with
> >> "test-stability" and mark them as "critical".
> >>
> >> What do you think about it?
> >>
> >>
> >> -Matthias
> >>
> >>
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Collecting Instable Tests

Márton Balassi
I like Fabian's approach, you can also share these filters on JIRA.

On Fri, Oct 9, 2015 at 2:55 PM, Fabian Hueske <[hidden email]> wrote:

> Sorry, that was not the correct link.
> You can create a bookmark for this one:
>
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC
>
> and directly ask for the list without searching.
>
> 2015-10-09 14:48 GMT+02:00 Matthias J. Sax <[hidden email]>:
>
> > Yes, but this filtering is super slow (or is it just my browser?)
> >
> > I personally don't like the search feature in JIRA and an umbrella would
> > give a easier to access overview.
> >
> > But if it's just me, never mind ;)
> >
> >
> > On 10/09/2015 02:39 PM, Fabian Hueske wrote:
> > > What would be the benefit over the current approach?
> > > You can just fiter JIRA for issues with a specific label:
> > >
> > >
> >
> https://issues.apache.org/jira/browse/FLINK-2771?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC
> > >
> > > Not sure if the gain (if there is any) justifies the overhead of adding
> > > issues to an umbrella issue.
> > >
> > >
> > > 2015-10-09 13:47 GMT+02:00 Matthias J. Sax <[hidden email]>:
> > >
> > >> Hi,
> > >>
> > >> right now we collect instable tests via labeling them with
> > >> "test-stability".
> > >>
> > >> I think it would be a better approach, to use an Umbrella Issue in
> JIRA
> > >> and collect them as Sub-Tasks. Storm does the same thing:
> > >> https://issues.apache.org/jira/browse/STORM-915
> > >>
> > >> This would have the advantage, that we get a simpler overview what
> tests
> > >> got already reported (and/or fixed). We can also list the tests by
> > >> component which makes it easier to find (the JIRA "search issues"
> > >> feature is cumbersome to use because the UI is quite slow)
> > >>
> > >> I would keep current approach and label the sub-tasks with
> > >> "test-stability" and mark them as "critical".
> > >>
> > >> What do you think about it?
> > >>
> > >>
> > >> -Matthias
> > >>
> > >>
> > >
> >
> >
>
mxm
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Collecting Instable Tests

mxm
In terms of usability, you might have a point Matthias. In terms of
JIRA workflow, I'd say that it makes more sense to tag and filter. An
issue with subtasks should be resolvable. Test instability, on the
other hand, requires a continuous effort and therefore not really
applicable to the concept of an issue. An "Umbrella" issue is not a
standard issue type and seems more like a hack to me.

On Fri, Oct 9, 2015 at 2:56 PM, Márton Balassi <[hidden email]> wrote:

> I like Fabian's approach, you can also share these filters on JIRA.
>
> On Fri, Oct 9, 2015 at 2:55 PM, Fabian Hueske <[hidden email]> wrote:
>
>> Sorry, that was not the correct link.
>> You can create a bookmark for this one:
>>
>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC
>>
>> and directly ask for the list without searching.
>>
>> 2015-10-09 14:48 GMT+02:00 Matthias J. Sax <[hidden email]>:
>>
>> > Yes, but this filtering is super slow (or is it just my browser?)
>> >
>> > I personally don't like the search feature in JIRA and an umbrella would
>> > give a easier to access overview.
>> >
>> > But if it's just me, never mind ;)
>> >
>> >
>> > On 10/09/2015 02:39 PM, Fabian Hueske wrote:
>> > > What would be the benefit over the current approach?
>> > > You can just fiter JIRA for issues with a specific label:
>> > >
>> > >
>> >
>> https://issues.apache.org/jira/browse/FLINK-2771?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20test-stability%20ORDER%20BY%20priority%20DESC
>> > >
>> > > Not sure if the gain (if there is any) justifies the overhead of adding
>> > > issues to an umbrella issue.
>> > >
>> > >
>> > > 2015-10-09 13:47 GMT+02:00 Matthias J. Sax <[hidden email]>:
>> > >
>> > >> Hi,
>> > >>
>> > >> right now we collect instable tests via labeling them with
>> > >> "test-stability".
>> > >>
>> > >> I think it would be a better approach, to use an Umbrella Issue in
>> JIRA
>> > >> and collect them as Sub-Tasks. Storm does the same thing:
>> > >> https://issues.apache.org/jira/browse/STORM-915
>> > >>
>> > >> This would have the advantage, that we get a simpler overview what
>> tests
>> > >> got already reported (and/or fixed). We can also list the tests by
>> > >> component which makes it easier to find (the JIRA "search issues"
>> > >> feature is cumbersome to use because the UI is quite slow)
>> > >>
>> > >> I would keep current approach and label the sub-tasks with
>> > >> "test-stability" and mark them as "critical".
>> > >>
>> > >> What do you think about it?
>> > >>
>> > >>
>> > >> -Matthias
>> > >>
>> > >>
>> > >
>> >
>> >
>>