(DEPRECATED) Apache Flink Mailing List archive.

[DISCUSS] Stop adding new bash-based e2e tests to Flink

Classic

List

Threaded

15 messages Options

Robert Metzger

[DISCUSS] Stop adding new bash-based e2e tests to Flink

Hi all,

Since we are currently testing the 1.12 release, and potentially adding
more automated e2e tests, I would like to bring up our end to end tests for
discussion.

Some time ago, we introduced a Java-based testing framework, with the idea
of replacing the current bash-based end to end tests.
Since the introduction of the java-based framework, more bash tests were
actually added, making a future migration even harder.

*For that reason, I would like to propose that we are stopping to add any
new bash end to end tests to Flink. All new end to end tests must be
written in Java and rely on the existing testing framework.*

For the 1.13 release, I'm trying to find some time to revisit potential
improvements for the existing java e2e framework (such as using Docker
images everywhere), as well as a migration plan for the existing bash
tests. We have a large number of bash e2e tests that are just parameterized
differently. If we would start migrating them to Java, we could move a
larger proportion of tests over to the new Java framework, and tackle the
more involved bash tests later (kerberized yarn, kubernetes, ...).

Let me know what you think!

Best,
Robert

PS: If you are wondering why I'm bringing this up now: I'm spending quite a
lot of time trying to figure out really hard to debug issues with our bash
testing infra.
Also, it is very difficult to introduce something generic for all tests
(such as a test-timeout, using docker as the preferred deployment method
etc.) since the tests often don't share common tooling.
Speaking about tooling: there are a lot of utilities everywhere, sometimes
duplicated, with different features / stability etc.
I believe bash is not the right tool for a project this size (in terms of
developers and lines of code)

Yang Wang

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Thanks for starting this discussion.

In general, I agree with you that a java-based testing framework is better
than the bash-based. It will
help a lot for the commons and utilities.

Since I am trying to add a new bash-based Kubernetes HA test, I have some
quick questions.
* I am not sure where the java-based framework is. Do you mean
"flink-jepsen" module or sth else?
* Maybe it will be harder to run a cli command(e.g. flink run /
run-application) to submit a Flink job in the java-based framework.
* It will be harder to inject some operations. For example, kill the
JobManager in Kubernetes. Currently, I
am trying to use "kubectl exec" to do this.

Best,
Yang

Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：

> Hi all,
>
> Since we are currently testing the 1.12 release, and potentially adding
> more automated e2e tests, I would like to bring up our end to end tests for
> discussion.
>
> Some time ago, we introduced a Java-based testing framework, with the idea
> of replacing the current bash-based end to end tests.
> Since the introduction of the java-based framework, more bash tests were
> actually added, making a future migration even harder.
>
> *For that reason, I would like to propose that we are stopping to add any
> new bash end to end tests to Flink. All new end to end tests must be
> written in Java and rely on the existing testing framework.*
>
> For the 1.13 release, I'm trying to find some time to revisit potential
> improvements for the existing java e2e framework (such as using Docker
> images everywhere), as well as a migration plan for the existing bash
> tests. We have a large number of bash e2e tests that are just parameterized
> differently. If we would start migrating them to Java, we could move a
> larger proportion of tests over to the new Java framework, and tackle the
> more involved bash tests later (kerberized yarn, kubernetes, ...).
>
> Let me know what you think!
>
> Best,
> Robert
>
>
> PS: If you are wondering why I'm bringing this up now: I'm spending quite a
> lot of time trying to figure out really hard to debug issues with our bash
> testing infra.
> Also, it is very difficult to introduce something generic for all tests
> (such as a test-timeout, using docker as the preferred deployment method
> etc.) since the tests often don't share common tooling.
> Speaking about tooling: there are a lot of utilities everywhere, sometimes
> duplicated, with different features / stability etc.
> I believe bash is not the right tool for a project this size (in terms of
> developers and lines of code)
>

Till Rohrmann

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Big +1 for your proposal Robert. I think the decision to use bash for our
e2e tests already fired back on us several times. Hence, I believe it is
time for a serious effort to get away from it.

The e2e test framework can be found here [1].

When using the e2e test framework, it should be possible to use Flink's
programmatic tools to submit a job (e.g. the RestClusterClient).

For killing K8s processes one could write a wrapper for kubectl which can
be called from the Java code. I think we already have it for some other cli
tools [2]. Alternatively, one could use Fabric8 to communicate with the K8s
cluster.

[1]
https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
[2]
https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/CommandLineWrapper.java

Cheers,
Till

On Wed, Nov 18, 2020 at 7:42 AM Yang Wang <[hidden email]> wrote:

> Thanks for starting this discussion.
>
> In general, I agree with you that a java-based testing framework is better
> than the bash-based. It will
> help a lot for the commons and utilities.
>
> Since I am trying to add a new bash-based Kubernetes HA test, I have some
> quick questions.
> * I am not sure where the java-based framework is. Do you mean
> "flink-jepsen" module or sth else?
> * Maybe it will be harder to run a cli command(e.g. flink run /
> run-application) to submit a Flink job in the java-based framework.
> * It will be harder to inject some operations. For example, kill the
> JobManager in Kubernetes. Currently, I
> am trying to use "kubectl exec" to do this.
>
>
> Best,
> Yang
>
> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
>
> > Hi all,
> >
> > Since we are currently testing the 1.12 release, and potentially adding
> > more automated e2e tests, I would like to bring up our end to end tests
> for
> > discussion.
> >
> > Some time ago, we introduced a Java-based testing framework, with the
> idea
> > of replacing the current bash-based end to end tests.
> > Since the introduction of the java-based framework, more bash tests were
> > actually added, making a future migration even harder.
> >
> > *For that reason, I would like to propose that we are stopping to add any
> > new bash end to end tests to Flink. All new end to end tests must be
> > written in Java and rely on the existing testing framework.*
> >
> > For the 1.13 release, I'm trying to find some time to revisit potential
> > improvements for the existing java e2e framework (such as using Docker
> > images everywhere), as well as a migration plan for the existing bash
> > tests. We have a large number of bash e2e tests that are just
> parameterized
> > differently. If we would start migrating them to Java, we could move a
> > larger proportion of tests over to the new Java framework, and tackle the
> > more involved bash tests later (kerberized yarn, kubernetes, ...).
> >
> > Let me know what you think!
> >
> > Best,
> > Robert
> >
> >
> > PS: If you are wondering why I'm bringing this up now: I'm spending
> quite a
> > lot of time trying to figure out really hard to debug issues with our
> bash
> > testing infra.
> > Also, it is very difficult to introduce something generic for all tests
> > (such as a test-timeout, using docker as the preferred deployment method
> > etc.) since the tests often don't share common tooling.
> > Speaking about tooling: there are a lot of utilities everywhere,
> sometimes
> > duplicated, with different features / stability etc.
> > I believe bash is not the right tool for a project this size (in terms of
> > developers and lines of code)
> >
>

Jark Wu-2

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

In reply to this post by Yang Wang

+1 to use the Java-based testing framework and +1 for using docker images
in the future.
IIUC, the Java-based testing framework refers to the
`flink-end-to-end-tests-common` module.
The java-based framework helped us a lot when debugging the unstable e2e
tests.

Best,
Jark

On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]> wrote:

Arvid Heise-3

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Also +1 for java-based testing framework; bash just disrupts development
flow too much.
+1 to use docker as much as possible, ideally with testcontainers [1].

[1] https://www.testcontainers.org/

On Wed, Nov 18, 2020 at 10:23 AM Jark Wu <[hidden email]> wrote:

> +1 to use the Java-based testing framework and +1 for using docker images
> in the future.
> IIUC, the Java-based testing framework refers to the
> `flink-end-to-end-tests-common` module.
> The java-based framework helped us a lot when debugging the unstable e2e
> tests.
>
> Best,
> Jark
>
> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]> wrote:
>
> > Thanks for starting this discussion.
> >
> > In general, I agree with you that a java-based testing framework is
> better
> > than the bash-based. It will
> > help a lot for the commons and utilities.
> >
> > Since I am trying to add a new bash-based Kubernetes HA test, I have some
> > quick questions.
> > * I am not sure where the java-based framework is. Do you mean
> > "flink-jepsen" module or sth else?
> > * Maybe it will be harder to run a cli command(e.g. flink run /
> > run-application) to submit a Flink job in the java-based framework.
> > * It will be harder to inject some operations. For example, kill the
> > JobManager in Kubernetes. Currently, I
> > am trying to use "kubectl exec" to do this.
> >
> >
> > Best,
> > Yang
> >
> > Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
> >
> > > Hi all,
> > >
> > > Since we are currently testing the 1.12 release, and potentially adding
> > > more automated e2e tests, I would like to bring up our end to end tests
> > for
> > > discussion.
> > >
> > > Some time ago, we introduced a Java-based testing framework, with the
> > idea
> > > of replacing the current bash-based end to end tests.
> > > Since the introduction of the java-based framework, more bash tests
> were
> > > actually added, making a future migration even harder.
> > >
> > > *For that reason, I would like to propose that we are stopping to add
> any
> > > new bash end to end tests to Flink. All new end to end tests must be
> > > written in Java and rely on the existing testing framework.*
> > >
> > > For the 1.13 release, I'm trying to find some time to revisit potential
> > > improvements for the existing java e2e framework (such as using Docker
> > > images everywhere), as well as a migration plan for the existing bash
> > > tests. We have a large number of bash e2e tests that are just
> > parameterized
> > > differently. If we would start migrating them to Java, we could move a
> > > larger proportion of tests over to the new Java framework, and tackle
> the
> > > more involved bash tests later (kerberized yarn, kubernetes, ...).
> > >
> > > Let me know what you think!
> > >
> > > Best,
> > > Robert
> > >
> > >
> > > PS: If you are wondering why I'm bringing this up now: I'm spending
> > quite a
> > > lot of time trying to figure out really hard to debug issues with our
> > bash
> > > testing infra.
> > > Also, it is very difficult to introduce something generic for all tests
> > > (such as a test-timeout, using docker as the preferred deployment
> method
> > > etc.) since the tests often don't share common tooling.
> > > Speaking about tooling: there are a lot of utilities everywhere,
> > sometimes
> > > duplicated, with different features / stability etc.
> > > I believe bash is not the right tool for a project this size (in terms
> of
> > > developers and lines of code)
> > >
> >
>

--

Arvid Heise | Senior Java Developer

<https://www.ververica.com/>

Follow us @VervericaData

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng

Yang Wang

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

In reply to this post by Jark Wu-2

Thanks till and Jark for sharing the information.

I am also +1 for this proposal and glad to wire the new introduced K8s HA
e2e tests to java based framework.

Best,
Yang

Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：

Aljoscha Krettek-2

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

+1

And I want to second Arvid's mention of testcontainers [1].

[1] https://www.testcontainers.org/

On 18.11.20 10:43, Yang Wang wrote:

> Thanks till and Jark for sharing the information.
>
> I am also +1 for this proposal and glad to wire the new introduced K8s HA
> e2e tests to java based framework.
>
> Best,
> Yang
>
> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
>
>> +1 to use the Java-based testing framework and +1 for using docker images
>> in the future.
>> IIUC, the Java-based testing framework refers to the
>> `flink-end-to-end-tests-common` module.
>> The java-based framework helped us a lot when debugging the unstable e2e
>> tests.
>>
>> Best,
>> Jark
>>
>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]> wrote:
>>
>>> Thanks for starting this discussion.
>>>
>>> In general, I agree with you that a java-based testing framework is
>> better
>>> than the bash-based. It will
>>> help a lot for the commons and utilities.
>>>
>>> Since I am trying to add a new bash-based Kubernetes HA test, I have some
>>> quick questions.
>>> * I am not sure where the java-based framework is. Do you mean
>>> "flink-jepsen" module or sth else?
>>> * Maybe it will be harder to run a cli command(e.g. flink run /
>>> run-application) to submit a Flink job in the java-based framework.
>>> * It will be harder to inject some operations. For example, kill the
>>> JobManager in Kubernetes. Currently, I
>>> am trying to use "kubectl exec" to do this.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
>>>
>>>> Hi all,
>>>>
>>>> Since we are currently testing the 1.12 release, and potentially adding
>>>> more automated e2e tests, I would like to bring up our end to end tests
>>> for
>>>> discussion.
>>>>
>>>> Some time ago, we introduced a Java-based testing framework, with the
>>> idea
>>>> of replacing the current bash-based end to end tests.
>>>> Since the introduction of the java-based framework, more bash tests
>> were
>>>> actually added, making a future migration even harder.
>>>>
>>>> *For that reason, I would like to propose that we are stopping to add
>> any
>>>> new bash end to end tests to Flink. All new end to end tests must be
>>>> written in Java and rely on the existing testing framework.*
>>>>
>>>> For the 1.13 release, I'm trying to find some time to revisit potential
>>>> improvements for the existing java e2e framework (such as using Docker
>>>> images everywhere), as well as a migration plan for the existing bash
>>>> tests. We have a large number of bash e2e tests that are just
>>> parameterized
>>>> differently. If we would start migrating them to Java, we could move a
>>>> larger proportion of tests over to the new Java framework, and tackle
>> the
>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
>>>>
>>>> Let me know what you think!
>>>>
>>>> Best,
>>>> Robert
>>>>
>>>>
>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
>>> quite a
>>>> lot of time trying to figure out really hard to debug issues with our
>>> bash
>>>> testing infra.
>>>> Also, it is very difficult to introduce something generic for all tests
>>>> (such as a test-timeout, using docker as the preferred deployment
>> method
>>>> etc.) since the tests often don't share common tooling.
>>>> Speaking about tooling: there are a lot of utilities everywhere,
>>> sometimes
>>>> duplicated, with different features / stability etc.
>>>> I believe bash is not the right tool for a project this size (in terms
>> of
>>>> developers and lines of code)
>>>>
>>>
>>
>

Leonard Xu

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

+1 to stop using bash scripts,
and I also have experienced the bash scripts that is really hard to maintain and debug, thanks @Robert for the great work again.

I think testcontainers is a nice candidate.

Best,
Leonard

> 在 2020年11月18日，19:46，Aljoscha Krettek <[hidden email]> 写道：
>
> +1
>
> And I want to second Arvid's mention of testcontainers [1].
>
> [1] https://www.testcontainers.org/
>
> On 18.11.20 10:43, Yang Wang wrote:
>> Thanks till and Jark for sharing the information.
>> I am also +1 for this proposal and glad to wire the new introduced K8s HA
>> e2e tests to java based framework.
>> Best,
>> Yang
>> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
>>> +1 to use the Java-based testing framework and +1 for using docker images
>>> in the future.
>>> IIUC, the Java-based testing framework refers to the
>>> `flink-end-to-end-tests-common` module.
>>> The java-based framework helped us a lot when debugging the unstable e2e
>>> tests.
>>>
>>> Best,
>>> Jark
>>>
>>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]> wrote:
>>>
>>>> Thanks for starting this discussion.
>>>>
>>>> In general, I agree with you that a java-based testing framework is
>>> better
>>>> than the bash-based. It will
>>>> help a lot for the commons and utilities.
>>>>
>>>> Since I am trying to add a new bash-based Kubernetes HA test, I have some
>>>> quick questions.
>>>> * I am not sure where the java-based framework is. Do you mean
>>>> "flink-jepsen" module or sth else?
>>>> * Maybe it will be harder to run a cli command(e.g. flink run /
>>>> run-application) to submit a Flink job in the java-based framework.
>>>> * It will be harder to inject some operations. For example, kill the
>>>> JobManager in Kubernetes. Currently, I
>>>> am trying to use "kubectl exec" to do this.
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
>>>>
>>>>> Hi all,
>>>>>
>>>>> Since we are currently testing the 1.12 release, and potentially adding
>>>>> more automated e2e tests, I would like to bring up our end to end tests
>>>> for
>>>>> discussion.
>>>>>
>>>>> Some time ago, we introduced a Java-based testing framework, with the
>>>> idea
>>>>> of replacing the current bash-based end to end tests.
>>>>> Since the introduction of the java-based framework, more bash tests
>>> were
>>>>> actually added, making a future migration even harder.
>>>>>
>>>>> *For that reason, I would like to propose that we are stopping to add
>>> any
>>>>> new bash end to end tests to Flink. All new end to end tests must be
>>>>> written in Java and rely on the existing testing framework.*
>>>>>
>>>>> For the 1.13 release, I'm trying to find some time to revisit potential
>>>>> improvements for the existing java e2e framework (such as using Docker
>>>>> images everywhere), as well as a migration plan for the existing bash
>>>>> tests. We have a large number of bash e2e tests that are just
>>>> parameterized
>>>>> differently. If we would start migrating them to Java, we could move a
>>>>> larger proportion of tests over to the new Java framework, and tackle
>>> the
>>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
>>>>>
>>>>> Let me know what you think!
>>>>>
>>>>> Best,
>>>>> Robert
>>>>>
>>>>>
>>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
>>>> quite a
>>>>> lot of time trying to figure out really hard to debug issues with our
>>>> bash
>>>>> testing infra.
>>>>> Also, it is very difficult to introduce something generic for all tests
>>>>> (such as a test-timeout, using docker as the preferred deployment
>>> method
>>>>> etc.) since the tests often don't share common tooling.
>>>>> Speaking about tooling: there are a lot of utilities everywhere,
>>>> sometimes
>>>>> duplicated, with different features / stability etc.
>>>>> I believe bash is not the right tool for a project this size (in terms
>>> of
>>>>> developers and lines of code)
>>>>>
>>>>
>>>
>

Rui Li

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Big +1 to java-based e2e tests. It'll be much easier to write/debug these
tests.

On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <[hidden email]> wrote:

> +1 to stop using bash scripts,
> and I also have experienced the bash scripts that is really hard to
> maintain and debug, thanks @Robert for the great work again.
>
> I think testcontainers is a nice candidate.
>
> Best,
> Leonard
>
> > 在 2020年11月18日，19:46，Aljoscha Krettek <[hidden email]> 写道：
> >
> > +1
> >
> > And I want to second Arvid's mention of testcontainers [1].
> >
> > [1] https://www.testcontainers.org/
> >
> > On 18.11.20 10:43, Yang Wang wrote:
> >> Thanks till and Jark for sharing the information.
> >> I am also +1 for this proposal and glad to wire the new introduced K8s
> HA
> >> e2e tests to java based framework.
> >> Best,
> >> Yang
> >> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
> >>> +1 to use the Java-based testing framework and +1 for using docker
> images
> >>> in the future.
> >>> IIUC, the Java-based testing framework refers to the
> >>> `flink-end-to-end-tests-common` module.
> >>> The java-based framework helped us a lot when debugging the unstable
> e2e
> >>> tests.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]> wrote:
> >>>
> >>>> Thanks for starting this discussion.
> >>>>
> >>>> In general, I agree with you that a java-based testing framework is
> >>> better
> >>>> than the bash-based. It will
> >>>> help a lot for the commons and utilities.
> >>>>
> >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have
> some
> >>>> quick questions.
> >>>> * I am not sure where the java-based framework is. Do you mean
> >>>> "flink-jepsen" module or sth else?
> >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> >>>> run-application) to submit a Flink job in the java-based framework.
> >>>> * It will be harder to inject some operations. For example, kill the
> >>>> JobManager in Kubernetes. Currently, I
> >>>> am trying to use "kubectl exec" to do this.
> >>>>
> >>>>
> >>>> Best,
> >>>> Yang
> >>>>
> >>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Since we are currently testing the 1.12 release, and potentially
> adding
> >>>>> more automated e2e tests, I would like to bring up our end to end
> tests
> >>>> for
> >>>>> discussion.
> >>>>>
> >>>>> Some time ago, we introduced a Java-based testing framework, with the
> >>>> idea
> >>>>> of replacing the current bash-based end to end tests.
> >>>>> Since the introduction of the java-based framework, more bash tests
> >>> were
> >>>>> actually added, making a future migration even harder.
> >>>>>
> >>>>> *For that reason, I would like to propose that we are stopping to add
> >>> any
> >>>>> new bash end to end tests to Flink. All new end to end tests must be
> >>>>> written in Java and rely on the existing testing framework.*
> >>>>>
> >>>>> For the 1.13 release, I'm trying to find some time to revisit
> potential
> >>>>> improvements for the existing java e2e framework (such as using
> Docker
> >>>>> images everywhere), as well as a migration plan for the existing bash
> >>>>> tests. We have a large number of bash e2e tests that are just
> >>>> parameterized
> >>>>> differently. If we would start migrating them to Java, we could move
> a
> >>>>> larger proportion of tests over to the new Java framework, and tackle
> >>> the
> >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
> >>>>>
> >>>>> Let me know what you think!
> >>>>>
> >>>>> Best,
> >>>>> Robert
> >>>>>
> >>>>>
> >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
> >>>> quite a
> >>>>> lot of time trying to figure out really hard to debug issues with our
> >>>> bash
> >>>>> testing infra.
> >>>>> Also, it is very difficult to introduce something generic for all
> tests
> >>>>> (such as a test-timeout, using docker as the preferred deployment
> >>> method
> >>>>> etc.) since the tests often don't share common tooling.
> >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> >>>> sometimes
> >>>>> duplicated, with different features / stability etc.
> >>>>> I believe bash is not the right tool for a project this size (in
> terms
> >>> of
> >>>>> developers and lines of code)
> >>>>>
> >>>>
> >>>
> >
>
>

--
Best regards!
Rui Li

Xingbo Huang

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Big +1 for java-based e2e tests. Currently PyFlink related tests each take
~15minutes in bash e2e tests because we are using a secured YARN cluster
which is the only convenient way of starting a YARN cluster in the bash e2e
tests. I think if we migrate these tests to the java-based testing
framework, we will start a Yarn Cluster more conveniently, which will
greatly reduce our testing time.

Best,
Xingbo

Rui Li <[hidden email]> 于2020年11月19日周四上午10:47写道：

> Big +1 to java-based e2e tests. It'll be much easier to write/debug these
> tests.
>
> On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <[hidden email]> wrote:
>
> > +1 to stop using bash scripts,
> > and I also have experienced the bash scripts that is really hard to
> > maintain and debug, thanks @Robert for the great work again.
> >
> > I think testcontainers is a nice candidate.
> >
> > Best,
> > Leonard
> >
> > > 在 2020年11月18日，19:46，Aljoscha Krettek <[hidden email]> 写道：
> > >
> > > +1
> > >
> > > And I want to second Arvid's mention of testcontainers [1].
> > >
> > > [1] https://www.testcontainers.org/
> > >
> > > On 18.11.20 10:43, Yang Wang wrote:
> > >> Thanks till and Jark for sharing the information.
> > >> I am also +1 for this proposal and glad to wire the new introduced K8s
> > HA
> > >> e2e tests to java based framework.
> > >> Best,
> > >> Yang
> > >> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
> > >>> +1 to use the Java-based testing framework and +1 for using docker
> > images
> > >>> in the future.
> > >>> IIUC, the Java-based testing framework refers to the
> > >>> `flink-end-to-end-tests-common` module.
> > >>> The java-based framework helped us a lot when debugging the unstable
> > e2e
> > >>> tests.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]>
> wrote:
> > >>>
> > >>>> Thanks for starting this discussion.
> > >>>>
> > >>>> In general, I agree with you that a java-based testing framework is
> > >>> better
> > >>>> than the bash-based. It will
> > >>>> help a lot for the commons and utilities.
> > >>>>
> > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have
> > some
> > >>>> quick questions.
> > >>>> * I am not sure where the java-based framework is. Do you mean
> > >>>> "flink-jepsen" module or sth else?
> > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > >>>> run-application) to submit a Flink job in the java-based framework.
> > >>>> * It will be harder to inject some operations. For example, kill the
> > >>>> JobManager in Kubernetes. Currently, I
> > >>>> am trying to use "kubectl exec" to do this.
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>> Yang
> > >>>>
> > >>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Since we are currently testing the 1.12 release, and potentially
> > adding
> > >>>>> more automated e2e tests, I would like to bring up our end to end
> > tests
> > >>>> for
> > >>>>> discussion.
> > >>>>>
> > >>>>> Some time ago, we introduced a Java-based testing framework, with
> the
> > >>>> idea
> > >>>>> of replacing the current bash-based end to end tests.
> > >>>>> Since the introduction of the java-based framework, more bash tests
> > >>> were
> > >>>>> actually added, making a future migration even harder.
> > >>>>>
> > >>>>> *For that reason, I would like to propose that we are stopping to
> add
> > >>> any
> > >>>>> new bash end to end tests to Flink. All new end to end tests must
> be
> > >>>>> written in Java and rely on the existing testing framework.*
> > >>>>>
> > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > potential
> > >>>>> improvements for the existing java e2e framework (such as using
> > Docker
> > >>>>> images everywhere), as well as a migration plan for the existing
> bash
> > >>>>> tests. We have a large number of bash e2e tests that are just
> > >>>> parameterized
> > >>>>> differently. If we would start migrating them to Java, we could
> move
> > a
> > >>>>> larger proportion of tests over to the new Java framework, and
> tackle
> > >>> the
> > >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
> > >>>>>
> > >>>>> Let me know what you think!
> > >>>>>
> > >>>>> Best,
> > >>>>> Robert
> > >>>>>
> > >>>>>
> > >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
> > >>>> quite a
> > >>>>> lot of time trying to figure out really hard to debug issues with
> our
> > >>>> bash
> > >>>>> testing infra.
> > >>>>> Also, it is very difficult to introduce something generic for all
> > tests
> > >>>>> (such as a test-timeout, using docker as the preferred deployment
> > >>> method
> > >>>>> etc.) since the tests often don't share common tooling.
> > >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> > >>>> sometimes
> > >>>>> duplicated, with different features / stability etc.
> > >>>>> I believe bash is not the right tool for a project this size (in
> > terms
> > >>> of
> > >>>>> developers and lines of code)
> > >>>>>
> > >>>>
> > >>>
> > >
> >
> >
>
> --
> Best regards!
> Rui Li
>

Yun Tang

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

+1 for java-based E2E test as bash scripts lack of the power to handle more complicated cases.

For the docker image improvement, I think we should be more cautious as developers at China might suffer with the network issue, or at least we should ensure or find some guides to speed up the image downloading.

Best
Yun Tang
________________________________
From: Xingbo Huang <[hidden email]>
Sent: Thursday, November 19, 2020 12:09
To: dev <[hidden email]>
Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Big +1 for java-based e2e tests. Currently PyFlink related tests each take
~15minutes in bash e2e tests because we are using a secured YARN cluster
which is the only convenient way of starting a YARN cluster in the bash e2e
tests. I think if we migrate these tests to the java-based testing
framework, we will start a Yarn Cluster more conveniently, which will
greatly reduce our testing time.

Best,
Xingbo

Rui Li <[hidden email]> 于2020年11月19日周四上午10:47写道：

Kevin Kwon

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

In reply to this post by Robert Metzger

Hi :)

For me, it's a bit unclear what it means by not using Docker in e2e tests
in K8S context, since K8S require Docker by default

Also, for for K8S test orchestration, wouldn't vanilla Python suffice? It
has all the K8S, Docker client libraries supported natively by them

Might just need to write a bit of health check scripts on other infras that
Flink supports like Kafka, S3 (mock) etc to safely start Flink and perform
any blackbox testing

On 2020/11/17 15:35:38, Robert Metzger <[hidden email]> wrote:
> Hi all,>
>
> Since we are currently testing the 1.12 release, and potentially adding>
> more automated e2e tests, I would like to bring up our end to end tests
for>
> discussion.>
>
> Some time ago, we introduced a Java-based testing framework, with the
idea>
> of replacing the current bash-based end to end tests.>
> Since the introduction of the java-based framework, more bash tests were>
> actually added, making a future migration even harder.>
>
> *For that reason, I would like to propose that we are stopping to add
any>
> new bash end to end tests to Flink. All new end to end tests must be>
> written in Java and rely on the existing testing framework.*>
>
> For the 1.13 release, I'm trying to find some time to revisit potential>
> improvements for the existing java e2e framework (such as using Docker>
> images everywhere), as well as a migration plan for the existing bash>
> tests. We have a large number of bash e2e tests that are just
parameterized>
> differently. If we would start migrating them to Java, we could move a>
> larger proportion of tests over to the new Java framework, and tackle
the>
> more involved bash tests later (kerberized yarn, kubernetes, ...).>
>
> Let me know what you think!>
>
> Best,>
> Robert>
>
>
> PS: If you are wondering why I'm bringing this up now: I'm spending quite
a>
> lot of time trying to figure out really hard to debug issues with our
bash>
> testing infra.>
> Also, it is very difficult to introduce something generic for all tests>
> (such as a test-timeout, using docker as the preferred deployment method>
> etc.) since the tests often don't share common tooling.>
> Speaking about tooling: there are a lot of utilities everywhere,
sometimes>
> duplicated, with different features / stability etc.>
> I believe bash is not the right tool for a project this size (in terms
of>
> developers and lines of code)>
>

--

*This email, including any information it contains and any
attachments to it, is confidential and may be privileged. This email is
intended only for the use of the named recipient(s). If you are not a named
recipient, please notify the sender immediately by replying to this message
and delete the original message. You should not disclose or copy this
email, any of its contents or any attachments to it. This email may have
been transmitted over an unsecure public network and, therefore, Careem
does not accept responsibility for its contents or for any damage sustained
as a result of viewing its contents or in connection with its transmission.
Careem reserves the right to monitor all communications from or to this
account.*

Matthias

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

In reply to this post by Yun Tang

Thanks Robert for pushing this. +1 for creating java-based e2e tests.

In the engine team, we decided to work towards the goal of migrating the
bash-based e2e tests to Java/Docker. We plan to migrate the existing
bash-based e2e tests located in the Engine team's component space
step-by-step. I created an umbrella Jira issue [1] to collect and document
the migration efforts. Feel free to do the same by creating subtasks under
[1].

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-20392

On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <[hidden email]> wrote:

> +1 for java-based E2E test as bash scripts lack of the power to handle
> more complicated cases.
>
> For the docker image improvement, I think we should be more cautious as
> developers at China might suffer with the network issue, or at least we
> should ensure or find some guides to speed up the image downloading.
>
> Best
> Yun Tang
> ________________________________
> From: Xingbo Huang <[hidden email]>
> Sent: Thursday, November 19, 2020 12:09
> To: dev <[hidden email]>
> Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
>
> Big +1 for java-based e2e tests. Currently PyFlink related tests each take
> ~15minutes in bash e2e tests because we are using a secured YARN cluster
> which is the only convenient way of starting a YARN cluster in the bash e2e
> tests. I think if we migrate these tests to the java-based testing
> framework, we will start a Yarn Cluster more conveniently, which will
> greatly reduce our testing time.
>
> Best,
> Xingbo
>
> Rui Li <[hidden email]> 于2020年11月19日周四上午10:47写道：
>
> > Big +1 to java-based e2e tests. It'll be much easier to write/debug these
> > tests.
> >
> > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <[hidden email]> wrote:
> >
> > > +1 to stop using bash scripts,
> > > and I also have experienced the bash scripts that is really hard to
> > > maintain and debug, thanks @Robert for the great work again.
> > >
> > > I think testcontainers is a nice candidate.
> > >
> > > Best,
> > > Leonard
> > >
> > > > 在 2020年11月18日，19:46，Aljoscha Krettek <[hidden email]> 写道：
> > > >
> > > > +1
> > > >
> > > > And I want to second Arvid's mention of testcontainers [1].
> > > >
> > > > [1] https://www.testcontainers.org/
> > > >
> > > > On 18.11.20 10:43, Yang Wang wrote:
> > > >> Thanks till and Jark for sharing the information.
> > > >> I am also +1 for this proposal and glad to wire the new introduced
> K8s
> > > HA
> > > >> e2e tests to java based framework.
> > > >> Best,
> > > >> Yang
> > > >> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
> > > >>> +1 to use the Java-based testing framework and +1 for using docker
> > > images
> > > >>> in the future.
> > > >>> IIUC, the Java-based testing framework refers to the
> > > >>> `flink-end-to-end-tests-common` module.
> > > >>> The java-based framework helped us a lot when debugging the
> unstable
> > > e2e
> > > >>> tests.
> > > >>>
> > > >>> Best,
> > > >>> Jark
> > > >>>
> > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]>
> > wrote:
> > > >>>
> > > >>>> Thanks for starting this discussion.
> > > >>>>
> > > >>>> In general, I agree with you that a java-based testing framework
> is
> > > >>> better
> > > >>>> than the bash-based. It will
> > > >>>> help a lot for the commons and utilities.
> > > >>>>
> > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I
> have
> > > some
> > > >>>> quick questions.
> > > >>>> * I am not sure where the java-based framework is. Do you mean
> > > >>>> "flink-jepsen" module or sth else?
> > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > > >>>> run-application) to submit a Flink job in the java-based
> framework.
> > > >>>> * It will be harder to inject some operations. For example, kill
> the
> > > >>>> JobManager in Kubernetes. Currently, I
> > > >>>> am trying to use "kubectl exec" to do this.
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>> Yang
> > > >>>>
> > > >>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
> > > >>>>
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Since we are currently testing the 1.12 release, and potentially
> > > adding
> > > >>>>> more automated e2e tests, I would like to bring up our end to end
> > > tests
> > > >>>> for
> > > >>>>> discussion.
> > > >>>>>
> > > >>>>> Some time ago, we introduced a Java-based testing framework, with
> > the
> > > >>>> idea
> > > >>>>> of replacing the current bash-based end to end tests.
> > > >>>>> Since the introduction of the java-based framework, more bash
> tests
> > > >>> were
> > > >>>>> actually added, making a future migration even harder.
> > > >>>>>
> > > >>>>> *For that reason, I would like to propose that we are stopping to
> > add
> > > >>> any
> > > >>>>> new bash end to end tests to Flink. All new end to end tests must
> > be
> > > >>>>> written in Java and rely on the existing testing framework.*
> > > >>>>>
> > > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > > potential
> > > >>>>> improvements for the existing java e2e framework (such as using
> > > Docker
> > > >>>>> images everywhere), as well as a migration plan for the existing
> > bash
> > > >>>>> tests. We have a large number of bash e2e tests that are just
> > > >>>> parameterized
> > > >>>>> differently. If we would start migrating them to Java, we could
> > move
> > > a
> > > >>>>> larger proportion of tests over to the new Java framework, and
> > tackle
> > > >>> the
> > > >>>>> more involved bash tests later (kerberized yarn, kubernetes,
> ...).
> > > >>>>>
> > > >>>>> Let me know what you think!
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Robert
> > > >>>>>
> > > >>>>>
> > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm
> spending
> > > >>>> quite a
> > > >>>>> lot of time trying to figure out really hard to debug issues with
> > our
> > > >>>> bash
> > > >>>>> testing infra.
> > > >>>>> Also, it is very difficult to introduce something generic for all
> > > tests
> > > >>>>> (such as a test-timeout, using docker as the preferred deployment
> > > >>> method
> > > >>>>> etc.) since the tests often don't share common tooling.
> > > >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> > > >>>> sometimes
> > > >>>>> duplicated, with different features / stability etc.
> > > >>>>> I believe bash is not the right tool for a project this size (in
> > > terms
> > > >>> of
> > > >>>>> developers and lines of code)
> > > >>>>>
> > > >>>>
> > > >>>
> > > >
> > >
> > >
> >
> > --
> > Best regards!
> > Rui Li
> >
>

--

Matthias Pohl | Engineer

Follow us @VervericaData Ververica <https://www.ververica.com/>

--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

Jark Wu-2

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Thanks Matthias,

I have created sub-tasks for the Table SQL related bash-based e2e tests.

Best,
Jark

On Fri, 27 Nov 2020 at 21:25, Matthias Pohl <[hidden email]> wrote:

> Thanks Robert for pushing this. +1 for creating java-based e2e tests.
>
> In the engine team, we decided to work towards the goal of migrating the
> bash-based e2e tests to Java/Docker. We plan to migrate the existing
> bash-based e2e tests located in the Engine team's component space
> step-by-step. I created an umbrella Jira issue [1] to collect and document
> the migration efforts. Feel free to do the same by creating subtasks under
> [1].
>
> Best,
> Matthias
>
> [1] https://issues.apache.org/jira/browse/FLINK-20392
>
> On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <[hidden email]> wrote:
>
> > +1 for java-based E2E test as bash scripts lack of the power to handle
> > more complicated cases.
> >
> > For the docker image improvement, I think we should be more cautious as
> > developers at China might suffer with the network issue, or at least we
> > should ensure or find some guides to speed up the image downloading.
> >
> > Best
> > Yun Tang
> > ________________________________
> > From: Xingbo Huang <[hidden email]>
> > Sent: Thursday, November 19, 2020 12:09
> > To: dev <[hidden email]>
> > Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
> >
> > Big +1 for java-based e2e tests. Currently PyFlink related tests each
> take
> > ~15minutes in bash e2e tests because we are using a secured YARN cluster
> > which is the only convenient way of starting a YARN cluster in the bash
> e2e
> > tests. I think if we migrate these tests to the java-based testing
> > framework, we will start a Yarn Cluster more conveniently, which will
> > greatly reduce our testing time.
> >
> > Best,
> > Xingbo
> >
> > Rui Li <[hidden email]> 于2020年11月19日周四上午10:47写道：
> >
> > > Big +1 to java-based e2e tests. It'll be much easier to write/debug
> these
> > > tests.
> > >
> > > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <[hidden email]> wrote:
> > >
> > > > +1 to stop using bash scripts,
> > > > and I also have experienced the bash scripts that is really hard to
> > > > maintain and debug, thanks @Robert for the great work again.
> > > >
> > > > I think testcontainers is a nice candidate.
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 在 2020年11月18日，19:46，Aljoscha Krettek <[hidden email]> 写道：
> > > > >
> > > > > +1
> > > > >
> > > > > And I want to second Arvid's mention of testcontainers [1].
> > > > >
> > > > > [1] https://www.testcontainers.org/
> > > > >
> > > > > On 18.11.20 10:43, Yang Wang wrote:
> > > > >> Thanks till and Jark for sharing the information.
> > > > >> I am also +1 for this proposal and glad to wire the new introduced
> > K8s
> > > > HA
> > > > >> e2e tests to java based framework.
> > > > >> Best,
> > > > >> Yang
> > > > >> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
> > > > >>> +1 to use the Java-based testing framework and +1 for using
> docker
> > > > images
> > > > >>> in the future.
> > > > >>> IIUC, the Java-based testing framework refers to the
> > > > >>> `flink-end-to-end-tests-common` module.
> > > > >>> The java-based framework helped us a lot when debugging the
> > unstable
> > > > e2e
> > > > >>> tests.
> > > > >>>
> > > > >>> Best,
> > > > >>> Jark
> > > > >>>
> > > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]>
> > > wrote:
> > > > >>>
> > > > >>>> Thanks for starting this discussion.
> > > > >>>>
> > > > >>>> In general, I agree with you that a java-based testing framework
> > is
> > > > >>> better
> > > > >>>> than the bash-based. It will
> > > > >>>> help a lot for the commons and utilities.
> > > > >>>>
> > > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I
> > have
> > > > some
> > > > >>>> quick questions.
> > > > >>>> * I am not sure where the java-based framework is. Do you mean
> > > > >>>> "flink-jepsen" module or sth else?
> > > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > > > >>>> run-application) to submit a Flink job in the java-based
> > framework.
> > > > >>>> * It will be harder to inject some operations. For example, kill
> > the
> > > > >>>> JobManager in Kubernetes. Currently, I
> > > > >>>> am trying to use "kubectl exec" to do this.
> > > > >>>>
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Yang
> > > > >>>>
> > > > >>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
> > > > >>>>
> > > > >>>>> Hi all,
> > > > >>>>>
> > > > >>>>> Since we are currently testing the 1.12 release, and
> potentially
> > > > adding
> > > > >>>>> more automated e2e tests, I would like to bring up our end to
> end
> > > > tests
> > > > >>>> for
> > > > >>>>> discussion.
> > > > >>>>>
> > > > >>>>> Some time ago, we introduced a Java-based testing framework,
> with
> > > the
> > > > >>>> idea
> > > > >>>>> of replacing the current bash-based end to end tests.
> > > > >>>>> Since the introduction of the java-based framework, more bash
> > tests
> > > > >>> were
> > > > >>>>> actually added, making a future migration even harder.
> > > > >>>>>
> > > > >>>>> *For that reason, I would like to propose that we are stopping
> to
> > > add
> > > > >>> any
> > > > >>>>> new bash end to end tests to Flink. All new end to end tests
> must
> > > be
> > > > >>>>> written in Java and rely on the existing testing framework.*
> > > > >>>>>
> > > > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > > > potential
> > > > >>>>> improvements for the existing java e2e framework (such as using
> > > > Docker
> > > > >>>>> images everywhere), as well as a migration plan for the
> existing
> > > bash
> > > > >>>>> tests. We have a large number of bash e2e tests that are just
> > > > >>>> parameterized
> > > > >>>>> differently. If we would start migrating them to Java, we could
> > > move
> > > > a
> > > > >>>>> larger proportion of tests over to the new Java framework, and
> > > tackle
> > > > >>> the
> > > > >>>>> more involved bash tests later (kerberized yarn, kubernetes,
> > ...).
> > > > >>>>>
> > > > >>>>> Let me know what you think!
> > > > >>>>>
> > > > >>>>> Best,
> > > > >>>>> Robert
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm
> > spending
> > > > >>>> quite a
> > > > >>>>> lot of time trying to figure out really hard to debug issues
> with
> > > our
> > > > >>>> bash
> > > > >>>>> testing infra.
> > > > >>>>> Also, it is very difficult to introduce something generic for
> all
> > > > tests
> > > > >>>>> (such as a test-timeout, using docker as the preferred
> deployment
> > > > >>> method
> > > > >>>>> etc.) since the tests often don't share common tooling.
> > > > >>>>> Speaking about tooling: there are a lot of utilities
> everywhere,
> > > > >>>> sometimes
> > > > >>>>> duplicated, with different features / stability etc.
> > > > >>>>> I believe bash is not the right tool for a project this size
> (in
> > > > terms
> > > > >>> of
> > > > >>>>> developers and lines of code)
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Best regards!
> > > Rui Li
> > >
> >
>
>
> --
>
> Matthias Pohl | Engineer
>
> Follow us @VervericaData Ververica <https://www.ververica.com/>
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
> Wehner
>

Xingbo Huang

Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink

Thanks Matthias,

I have created sub-tasks for PyFlink related e2e tests.

Best,
Xingbo

> 2020年11月27日下午10:53，Jark Wu <[hidden email]> 写道：
>
> Thanks Matthias,
>
> I have created sub-tasks for the Table SQL related bash-based e2e tests.
>
> Best,
> Jark
>
> On Fri, 27 Nov 2020 at 21:25, Matthias Pohl <[hidden email]> wrote:
>
>> Thanks Robert for pushing this. +1 for creating java-based e2e tests.
>>
>> In the engine team, we decided to work towards the goal of migrating the
>> bash-based e2e tests to Java/Docker. We plan to migrate the existing
>> bash-based e2e tests located in the Engine team's component space
>> step-by-step. I created an umbrella Jira issue [1] to collect and document
>> the migration efforts. Feel free to do the same by creating subtasks under
>> [1].
>>
>> Best,
>> Matthias
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-20392
>>
>> On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <[hidden email]> wrote:
>>
>>> +1 for java-based E2E test as bash scripts lack of the power to handle
>>> more complicated cases.
>>>
>>> For the docker image improvement, I think we should be more cautious as
>>> developers at China might suffer with the network issue, or at least we
>>> should ensure or find some guides to speed up the image downloading.
>>>
>>> Best
>>> Yun Tang
>>> ________________________________
>>> From: Xingbo Huang <[hidden email]>
>>> Sent: Thursday, November 19, 2020 12:09
>>> To: dev <[hidden email]>
>>> Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
>>>
>>> Big +1 for java-based e2e tests. Currently PyFlink related tests each
>> take
>>> ~15minutes in bash e2e tests because we are using a secured YARN cluster
>>> which is the only convenient way of starting a YARN cluster in the bash
>> e2e
>>> tests. I think if we migrate these tests to the java-based testing
>>> framework, we will start a Yarn Cluster more conveniently, which will
>>> greatly reduce our testing time.
>>>
>>> Best,
>>> Xingbo
>>>
>>> Rui Li <[hidden email]> 于2020年11月19日周四上午10:47写道：
>>>
>>>> Big +1 to java-based e2e tests. It'll be much easier to write/debug
>> these
>>>> tests.
>>>>
>>>> On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <[hidden email]> wrote:
>>>>
>>>>> +1 to stop using bash scripts,
>>>>> and I also have experienced the bash scripts that is really hard to
>>>>> maintain and debug, thanks @Robert for the great work again.
>>>>>
>>>>> I think testcontainers is a nice candidate.
>>>>>
>>>>> Best,
>>>>> Leonard
>>>>>
>>>>>> 在 2020年11月18日，19:46，Aljoscha Krettek <[hidden email]> 写道：
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> And I want to second Arvid's mention of testcontainers [1].
>>>>>>
>>>>>> [1] https://www.testcontainers.org/
>>>>>>
>>>>>> On 18.11.20 10:43, Yang Wang wrote:
>>>>>>> Thanks till and Jark for sharing the information.
>>>>>>> I am also +1 for this proposal and glad to wire the new introduced
>>> K8s
>>>>> HA
>>>>>>> e2e tests to java based framework.
>>>>>>> Best,
>>>>>>> Yang
>>>>>>> Jark Wu <[hidden email]> 于2020年11月18日周三下午5:23写道：
>>>>>>>> +1 to use the Java-based testing framework and +1 for using
>> docker
>>>>> images
>>>>>>>> in the future.
>>>>>>>> IIUC, the Java-based testing framework refers to the
>>>>>>>> `flink-end-to-end-tests-common` module.
>>>>>>>> The java-based framework helped us a lot when debugging the
>>> unstable
>>>>> e2e
>>>>>>>> tests.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Jark
>>>>>>>>
>>>>>>>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <[hidden email]>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>
>>>>>>>>> In general, I agree with you that a java-based testing framework
>>> is
>>>>>>>> better
>>>>>>>>> than the bash-based. It will
>>>>>>>>> help a lot for the commons and utilities.
>>>>>>>>>
>>>>>>>>> Since I am trying to add a new bash-based Kubernetes HA test, I
>>> have
>>>>> some
>>>>>>>>> quick questions.
>>>>>>>>> * I am not sure where the java-based framework is. Do you mean
>>>>>>>>> "flink-jepsen" module or sth else?
>>>>>>>>> * Maybe it will be harder to run a cli command(e.g. flink run /
>>>>>>>>> run-application) to submit a Flink job in the java-based
>>> framework.
>>>>>>>>> * It will be harder to inject some operations. For example, kill
>>> the
>>>>>>>>> JobManager in Kubernetes. Currently, I
>>>>>>>>> am trying to use "kubectl exec" to do this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Yang
>>>>>>>>>
>>>>>>>>> Robert Metzger <[hidden email]> 于2020年11月17日周二下午11:36写道：
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Since we are currently testing the 1.12 release, and
>> potentially
>>>>> adding
>>>>>>>>>> more automated e2e tests, I would like to bring up our end to
>> end
>>>>> tests
>>>>>>>>> for
>>>>>>>>>> discussion.
>>>>>>>>>>
>>>>>>>>>> Some time ago, we introduced a Java-based testing framework,
>> with
>>>> the
>>>>>>>>> idea
>>>>>>>>>> of replacing the current bash-based end to end tests.
>>>>>>>>>> Since the introduction of the java-based framework, more bash
>>> tests
>>>>>>>> were
>>>>>>>>>> actually added, making a future migration even harder.
>>>>>>>>>>
>>>>>>>>>> *For that reason, I would like to propose that we are stopping
>> to
>>>> add
>>>>>>>> any
>>>>>>>>>> new bash end to end tests to Flink. All new end to end tests
>> must
>>>> be
>>>>>>>>>> written in Java and rely on the existing testing framework.*
>>>>>>>>>>
>>>>>>>>>> For the 1.13 release, I'm trying to find some time to revisit
>>>>> potential
>>>>>>>>>> improvements for the existing java e2e framework (such as using
>>>>> Docker
>>>>>>>>>> images everywhere), as well as a migration plan for the
>> existing
>>>> bash
>>>>>>>>>> tests. We have a large number of bash e2e tests that are just
>>>>>>>>> parameterized
>>>>>>>>>> differently. If we would start migrating them to Java, we could
>>>> move
>>>>> a
>>>>>>>>>> larger proportion of tests over to the new Java framework, and
>>>> tackle
>>>>>>>> the
>>>>>>>>>> more involved bash tests later (kerberized yarn, kubernetes,
>>> ...).
>>>>>>>>>>
>>>>>>>>>> Let me know what you think!
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Robert
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> PS: If you are wondering why I'm bringing this up now: I'm
>>> spending
>>>>>>>>> quite a
>>>>>>>>>> lot of time trying to figure out really hard to debug issues
>> with
>>>> our
>>>>>>>>> bash
>>>>>>>>>> testing infra.
>>>>>>>>>> Also, it is very difficult to introduce something generic for
>> all
>>>>> tests
>>>>>>>>>> (such as a test-timeout, using docker as the preferred
>> deployment
>>>>>>>> method
>>>>>>>>>> etc.) since the tests often don't share common tooling.
>>>>>>>>>> Speaking about tooling: there are a lot of utilities
>> everywhere,
>>>>>>>>> sometimes
>>>>>>>>>> duplicated, with different features / stability etc.
>>>>>>>>>> I believe bash is not the right tool for a project this size
>> (in
>>>>> terms
>>>>>>>> of
>>>>>>>>>> developers and lines of code)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Best regards!
>>>> Rui Li
>>>>
>>>
>>
>>
>> --
>>
>> Matthias Pohl | Engineer
>>
>> Follow us @VervericaData Ververica <https://www.ververica.com/>
>>
>> --
>>
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
>> Wehner
>>