How to run k8s-related end-to-end tests locally?
Locked
 
|
Hi,
I tried to run single k8s e-2-e test locally on my macos laptop (a minikube vm has been started manually), and here is the command I run from the source root. ``` $ flink-end-to-end-tests/run-single-test.sh flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh ``` Then it failed at the image build stage while downloading the flink.tgz file from a local fileserver. ``` Step 10/14 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in ha.pool.sks-keyservers.net $(shuf -e hkp:// p80.pool.sks-keyservers.net:80 keyserver.ubuntu.com hkp:// keyserver.ubuntu.com:80 pgp.mit.edu) ; do gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink .; ---> Running in 2225f4ea3406 + wget -nv -O flink.tgz localhost:9999/flink.tgz failed: Connection refused. The command '/bin/sh -c set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in ha.pool.sks-keyservers.net $(shuf -e hkp:// p80.pool.sks-keyservers.net:80 keyserver.ubuntu.com hkp:// keyserver.ubuntu.com:80 pgp.mit.edu) ; do gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink .;' returned a non-zero code: 4 ``` I checked the source code and found the python fileserver is here: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L44 Then it uses 'docker build --net==host' to build image: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L53 I am confused here, --net=host only makes services at the minikube vm available to the docker build process, however, the python fileserver is running on my local env rather than the minikube vm, how the docker build process access that fileserver? Should I: - run the test script inside the minikube vm (so that the fileserver will run on it), or - did I miss some configuration to my minikube vm? Thanks in advance. |
|
|
Hi Xu Di,
Sorry for the inconvenience. This is a known issue FLINK-17857[1]. After we migrate all the e2e tests from flink-container/docker to apache/flink-docker, the docker and k8s related e2e tests could not work. I will revive this ticket and fix it ASAP. [1]. https://issues.apache.org/jira/browse/FLINK-17857 Best, Yang Di Xu <[hidden email]> 于2020年9月23日周三 下午4:54写道： > Hi, > > I tried to run single k8s e-2-e test locally on my macos laptop (a minikube > vm has been started manually), and here is the command I run from the > source root. > > ``` > $ flink-end-to-end-tests/run-single-test.sh > flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh > ``` > Then it failed at the image build stage while downloading the flink.tgz > file from a local fileserver. > > ``` > Step 10/14 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; if > [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc > "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in > ha.pool.sks-keyservers.net $(shuf -e hkp:// > p80.pool.sks-keyservers.net:80 > keyserver.ubuntu.com hkp:// > keyserver.ubuntu.com:80 pgp.mit.edu) ; do > gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; > done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf > --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf > flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink .; > ---> Running in 2225f4ea3406 > + wget -nv -O flink.tgz localhost:9999/flink.tgz > failed: Connection refused. > The command '/bin/sh -c set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; > if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc > "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in > ha.pool.sks-keyservers.net $(shuf -e hkp:// > p80.pool.sks-keyservers.net:80 > keyserver.ubuntu.com hkp:// > keyserver.ubuntu.com:80 pgp.mit.edu) ; do > gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : ; > done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf > --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf > flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink > .;' returned a non-zero code: 4 > ``` > > I checked the source code and found the python fileserver is here: > > > https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L44 > > Then it uses 'docker build --net==host' to build image: > > > https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L53 > > I am confused here, --net=host only makes services at the minikube vm > available to the docker build process, however, the python fileserver is > running on my local env rather than the minikube vm, how the docker build > process access that fileserver? > > Should I: > - run the test script inside the minikube vm (so that the fileserver will > run on it), or > - did I miss some configuration to my minikube vm? > > Thanks in advance. > |
|
|
Hi Wang Yang，
Thanks for your quick reply. I followed the discussion in FLINK-17857 and now understand the problem and solution. I will fix it locally and please let me know if there is anything I can help with this issue. Yang Wang <[hidden email]>于2020年9月24日 周四上午12:12写道： > Hi Xu Di, > > Sorry for the inconvenience. This is a known issue FLINK-17857[1]. After we > migrate all the e2e tests > from flink-container/docker to apache/flink-docker, the docker and k8s > related e2e tests could not work. > I will revive this ticket and fix it ASAP. > > [1]. https://issues.apache.org/jira/browse/FLINK-17857 > > Best, > Yang > > > Di Xu <[hidden email]> 于2020年9月23日周三 下午4:54写道： > > > Hi, > > > > I tried to run single k8s e-2-e test locally on my macos laptop (a > minikube > > vm has been started manually), and here is the command I run from the > > source root. > > > > ``` > > $ flink-end-to-end-tests/run-single-test.sh > > flink-end-to-end-tests/test-scripts/test_kubernetes_application.sh > > ``` > > Then it failed at the image build stage while downloading the flink.tgz > > file from a local fileserver. > > > > ``` > > Step 10/14 : RUN set -ex; wget -nv -O flink.tgz "$FLINK_TGZ_URL"; > if > > [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc > > "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in > > ha.pool.sks-keyservers.net $(shuf -e hkp:// > > p80.pool.sks-keyservers.net:80 > > keyserver.ubuntu.com hkp:// > > keyserver.ubuntu.com:80 pgp.mit.edu) ; do > > gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : > ; > > done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf > > --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf > > flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink > .; > > ---> Running in 2225f4ea3406 > > + wget -nv -O flink.tgz localhost:9999/flink.tgz > > failed: Connection refused. > > The command '/bin/sh -c set -ex; wget -nv -O flink.tgz > "$FLINK_TGZ_URL"; > > if [ "$CHECK_GPG" = "true" ]; then wget -nv -O flink.tgz.asc > > "$FLINK_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; for server in > > ha.pool.sks-keyservers.net $(shuf -e hkp:// > > p80.pool.sks-keyservers.net:80 > > keyserver.ubuntu.com hkp:// > > keyserver.ubuntu.com:80 pgp.mit.edu) ; do > > gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break || : > ; > > done && gpg --batch --verify flink.tgz.asc flink.tgz; gpgconf > > --kill all; rm -rf "$GNUPGHOME" flink.tgz.asc; fi; tar -xf > > flink.tgz --strip-components=1; rm flink.tgz; chown -R flink:flink > > .;' returned a non-zero code: 4 > > ``` > > > > I checked the source code and found the python fileserver is here: > > > > > > > https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L44 > > > > Then it uses 'docker build --net==host' to build image: > > > > > > > https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L53 > > > > I am confused here, --net=host only makes services at the minikube vm > > available to the docker build process, however, the python fileserver is > > running on my local env rather than the minikube vm, how the docker build > > process access that fileserver? > > > > Should I: > > - run the test script inside the minikube vm (so that the fileserver will > > run on it), or > > - did I miss some configuration to my minikube vm? > > > > Thanks in advance. > > > |
«
Return to (DEPRECATED) Apache Flink Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |