Add a module for "manual" tests

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Add a module for "manual" tests

Stephan Ewen
Hi all!

We have by now quite some tests that are not really suitable for execution
as part of the regular build process.

  - Some tests just take very long, for example some of the sorter tests
with multiple recursive merges.

  - Some tests depend on external components. These components are
sometimes not stable enough for reliable tests. An example is the
ElasticSearchITCase

How about we add a dedicated module for these tests, that we execute during
release testing, but not as part of the regular build tests?

Greetings,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: Add a module for "manual" tests

Till Rohrmann
+1 for your proposal Stephan. Seems like a good idea to improve build
stability as well as guarding components like the ElasticSearchSink by a
test, even though it’s not regularly executed.


On Tue, Sep 29, 2015 at 12:34 PM, Stephan Ewen <[hidden email]> wrote:

> Hi all!
>
> We have by now quite some tests that are not really suitable for execution
> as part of the regular build process.
>
>   - Some tests just take very long, for example some of the sorter tests
> with multiple recursive merges.
>
>   - Some tests depend on external components. These components are
> sometimes not stable enough for reliable tests. An example is the
> ElasticSearchITCase
>
> How about we add a dedicated module for these tests, that we execute during
> release testing, but not as part of the regular build tests?
>
> Greetings,
> Stephan
>
Reply | Threaded
Open this post in threaded view
|

Re: Add a module for "manual" tests

Robert Metzger
I agree that some tests are pretty resource intensive and/or unstable...

However I fear that we are ending up fixing our connectors for every
release (potentially delaying releases and forcing people to fix tests for
changes they didn't introduce)
We could try out adding another Travis build which is only executing those
heavy tests...



On Tue, Sep 29, 2015 at 1:40 PM, Till Rohrmann <[hidden email]> wrote:

> +1 for your proposal Stephan. Seems like a good idea to improve build
> stability as well as guarding components like the ElasticSearchSink by a
> test, even though it’s not regularly executed.
> ​
>
> On Tue, Sep 29, 2015 at 12:34 PM, Stephan Ewen <[hidden email]> wrote:
>
> > Hi all!
> >
> > We have by now quite some tests that are not really suitable for
> execution
> > as part of the regular build process.
> >
> >   - Some tests just take very long, for example some of the sorter tests
> > with multiple recursive merges.
> >
> >   - Some tests depend on external components. These components are
> > sometimes not stable enough for reliable tests. An example is the
> > ElasticSearchITCase
> >
> > How about we add a dedicated module for these tests, that we execute
> during
> > release testing, but not as part of the regular build tests?
> >
> > Greetings,
> > Stephan
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Add a module for "manual" tests

Stephan Ewen
@Robert: This does not mean that we move all connector tests to that
package, but only the ones that we cannot fully control (like ElasticSearch)

On Tue, Sep 29, 2015 at 3:49 PM, Robert Metzger <[hidden email]> wrote:

> I agree that some tests are pretty resource intensive and/or unstable...
>
> However I fear that we are ending up fixing our connectors for every
> release (potentially delaying releases and forcing people to fix tests for
> changes they didn't introduce)
> We could try out adding another Travis build which is only executing those
> heavy tests...
>
>
>
> On Tue, Sep 29, 2015 at 1:40 PM, Till Rohrmann <[hidden email]>
> wrote:
>
> > +1 for your proposal Stephan. Seems like a good idea to improve build
> > stability as well as guarding components like the ElasticSearchSink by a
> > test, even though it’s not regularly executed.
> > ​
> >
> > On Tue, Sep 29, 2015 at 12:34 PM, Stephan Ewen <[hidden email]> wrote:
> >
> > > Hi all!
> > >
> > > We have by now quite some tests that are not really suitable for
> > execution
> > > as part of the regular build process.
> > >
> > >   - Some tests just take very long, for example some of the sorter
> tests
> > > with multiple recursive merges.
> > >
> > >   - Some tests depend on external components. These components are
> > > sometimes not stable enough for reliable tests. An example is the
> > > ElasticSearchITCase
> > >
> > > How about we add a dedicated module for these tests, that we execute
> > during
> > > release testing, but not as part of the regular build tests?
> > >
> > > Greetings,
> > > Stephan
> > >
> >
>