[HowTo] Testing Flink Playground (was: [ANNOUNCE] Progress updates for Apache Flink 1.9.0 release)

[Fabian Hueske-2]

[Forking off this thread to keep the announce thread "clean"]

Hi Kurt,

The playground needs a bit of manual work at the moment, because 1.9 is not
released yet.
The docker-compose and Flink configurations are still in a PR [1].
Also the Flink 1.9 Docker containers need to manually build. When 1.9 is
out, we'll publish the container to Docker Hub to make it available as for
the other Flink versions before.

Instructions to build the container:

1) backport commit f695a76b10b0cb5f074bbb874fe374cd11e6eff3 to the
release-1.9 branch
2) Build the release-1.9 branch
3) clone this repository: https://github.com/docker-flink/docker-flink
4) copy the 1.9 build-target folder to docker-flink/1.8/scala_2.11-debian
5) change Dockerfile to the content below [2] (between START Dockerfile and
END Dockerfile)
6) Build the Docker container with docker build -t
flink:1.9-SNAPSHOT-scala_2.11 .

The container is now available and you can follow the steps in the
playground.

Best, Fabian

[1] https://github.com/apache/flink-playgrounds/pull/1
[2]
### START Dockerfile
###############################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################

FROM openjdk:8-jre

# Install dependencies
RUN set -ex; \
  apt-get update; \
  apt-get -y install libsnappy1v5; \
  rm -rf /var/lib/apt/lists/*

# Grab gosu for easy step-down from root
ENV GOSU_VERSION 1.11
RUN set -ex; \
  wget -nv -O /usr/local/bin/gosu "
https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
--print-architecture)"; \
  wget -nv -O /usr/local/bin/gosu.asc "
https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
--print-architecture).asc"; \
  export GNUPGHOME="$(mktemp -d)"; \
  for server in ha.pool.sks-keyservers.net $(shuf -e \
                          hkp://p80.pool.sks-keyservers.net:80 \
                          keyserver.ubuntu.com \
                          hkp://keyserver.ubuntu.com:80 \
                          pgp.mit.edu) ; do \
      gpg --batch --keyserver "$server" --recv-keys
B42F6819007F00F88E364FD4036A9C25BF357DD4 && break || : ; \
  done && \
  gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
  gpgconf --kill all; \
  rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \
  chmod +x /usr/local/bin/gosu; \
  gosu nobody true

# Configure Flink version
ENV FLINK_VERSION=1.8.1 \
    HADOOP_SCALA_VARIANT=scala_2.11 \
    GPG_KEY=8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E

# Prepare environment
ENV FLINK_HOME=/opt/flink
ENV PATH=$FLINK_HOME/bin:$PATH
RUN groupadd --system --gid=9999 flink && \
    useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=flink flink
WORKDIR $FLINK_HOME

ENV
FLINK_URL_FILE_PATH=flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-${HADOOP_SCALA_VARIANT}.tgz
# Not all mirrors have the .asc files
ENV FLINK_TGZ_URL=
https://www.apache.org/dyn/closer.cgi?action=download&filename=${FLINK_URL_FILE_PATH}
\
    FLINK_ASC_URL=https://www.apache.org/dist/${FLINK_URL_FILE_PATH}.asc

# Install Flink
#RUN set -ex; \
#  wget -nv -O flink.tgz "$FLINK_TGZ_URL"; \
#  wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; \
#  \
#  export GNUPGHOME="$(mktemp -d)"; \
#  for server in ha.pool.sks-keyservers.net $(shuf -e \
#                          hkp://p80.pool.sks-keyservers.net:80 \
#                          keyserver.ubuntu.com \
#                          hkp://keyserver.ubuntu.com:80 \
#                          pgp.mit.edu) ; do \
#      gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break ||
: ; \
#  done && \
#  gpg --batch --verify flink.tgz.asc flink.tgz; \
#  gpgconf --kill all; \
#  rm -rf "$GNUPGHOME" flink.tgz.asc; \
#  \
#  tar -xf flink.tgz --strip-components=1; \
#  rm flink.tgz; \
#  \


COPY build-target /opt/flink
RUN  chown -R flink:flink $FLINK_HOME;

# Configure container
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["help"]
### END Dockerfile

Am Do., 8. Aug. 2019 um 09:23 Uhr schrieb Kurt Young <[hidden email]>:

> +1 to include this in 1.9.0, adding some examples doesn't look like new
> feature to me.
> BTW, I am also trying this tutorial based on release-1.9 branch, but
> blocked by:
>
> git clone --branch release-1.10-SNAPSHOT
> [hidden email]:apache/flink-playgrounds.git
>
> Neither 1.10 nor 1.9 exists in flink-playground yet.
>
> Best,
> Kurt
>
>
> On Thu, Aug 8, 2019 at 3:18 PM Fabian Hueske <[hidden email]> wrote:
>
> > Hi,
> > I worked with Konstantin and reviewed the PR.
> > I think the playground is a great way to get started with Flink and
> explore
> > it's recovery mechanism and unique features like savepoints.
> >
> > I'm in favor of adding the required streaming example program for the 1.9
> > release unless there's a good technical argument against it.
> >
> > Best, Fabian
> >
>

[Kurt Young]

Thanks for the detailed instructions!

Best,
Kurt


On Thu, Aug 8, 2019 at 3:40 PM Fabian Hueske <[hidden email]> wrote:

> [Forking off this thread to keep the announce thread "clean"]
>
> Hi Kurt,
>
> The playground needs a bit of manual work at the moment, because 1.9 is
> not released yet.
> The docker-compose and Flink configurations are still in a PR [1].
> Also the Flink 1.9 Docker containers need to manually build. When 1.9 is
> out, we'll publish the container to Docker Hub to make it available as for
> the other Flink versions before.
>
> Instructions to build the container:
>
> 1) backport commit f695a76b10b0cb5f074bbb874fe374cd11e6eff3 to the
> release-1.9 branch
> 2) Build the release-1.9 branch
> 3) clone this repository: https://github.com/docker-flink/docker-flink
> 4) copy the 1.9 build-target folder to docker-flink/1.8/scala_2.11-debian
> 5) change Dockerfile to the content below [2] (between START Dockerfile
> and END Dockerfile)
> 6) Build the Docker container with docker build -t
> flink:1.9-SNAPSHOT-scala_2.11 .
>
> The container is now available and you can follow the steps in the
> playground.
>
> Best, Fabian
>
> [1] https://github.com/apache/flink-playgrounds/pull/1
> [2]
> ### START Dockerfile
>
> ###############################################################################
> #  Licensed to the Apache Software Foundation (ASF) under one
> #  or more contributor license agreements.  See the NOTICE file
> #  distributed with this work for additional information
> #  regarding copyright ownership.  The ASF licenses this file
> #  to you under the Apache License, Version 2.0 (the
> #  "License"); you may not use this file except in compliance
> #  with the License.  You may obtain a copy of the License at
> #
> #      http://www.apache.org/licenses/LICENSE-2.0
> #
> #  Unless required by applicable law or agreed to in writing, software
> #  distributed under the License is distributed on an "AS IS" BASIS,
> #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> #  See the License for the specific language governing permissions and
> # limitations under the License.
>
> ###############################################################################
>
> FROM openjdk:8-jre
>
> # Install dependencies
> RUN set -ex; \
>   apt-get update; \
>   apt-get -y install libsnappy1v5; \
>   rm -rf /var/lib/apt/lists/*
>
> # Grab gosu for easy step-down from root
> ENV GOSU_VERSION 1.11
> RUN set -ex; \
>   wget -nv -O /usr/local/bin/gosu "
> https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
> --print-architecture)"; \
>   wget -nv -O /usr/local/bin/gosu.asc "
> https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
> --print-architecture).asc"; \
>   export GNUPGHOME="$(mktemp -d)"; \
>   for server in ha.pool.sks-keyservers.net $(shuf -e \
>                           hkp://p80.pool.sks-keyservers.net:80 \
>                           keyserver.ubuntu.com \
>                           hkp://keyserver.ubuntu.com:80 \
>                           pgp.mit.edu) ; do \
>       gpg --batch --keyserver "$server" --recv-keys
> B42F6819007F00F88E364FD4036A9C25BF357DD4 && break || : ; \
>   done && \
>   gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \
>   gpgconf --kill all; \
>   rm -rf "$GNUPGHOME" /usr/local/bin/gosu.asc; \
>   chmod +x /usr/local/bin/gosu; \
>   gosu nobody true
>
> # Configure Flink version
> ENV FLINK_VERSION=1.8.1 \
>     HADOOP_SCALA_VARIANT=scala_2.11 \
>     GPG_KEY=8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E
>
> # Prepare environment
> ENV FLINK_HOME=/opt/flink
> ENV PATH=$FLINK_HOME/bin:$PATH
> RUN groupadd --system --gid=9999 flink && \
>     useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=flink flink
> WORKDIR $FLINK_HOME
>
> ENV
> FLINK_URL_FILE_PATH=flink/flink-${FLINK_VERSION}/flink-${FLINK_VERSION}-bin-${HADOOP_SCALA_VARIANT}.tgz
> # Not all mirrors have the .asc files
> ENV FLINK_TGZ_URL=
> https://www.apache.org/dyn/closer.cgi?action=download&filename=${FLINK_URL_FILE_PATH}
> \
>     FLINK_ASC_URL=https://www.apache.org/dist/${FLINK_URL_FILE_PATH}.asc
>
> # Install Flink
> #RUN set -ex; \
> #  wget -nv -O flink.tgz "$FLINK_TGZ_URL"; \
> #  wget -nv -O flink.tgz.asc "$FLINK_ASC_URL"; \
> #  \
> #  export GNUPGHOME="$(mktemp -d)"; \
> #  for server in ha.pool.sks-keyservers.net $(shuf -e \
> #                          hkp://p80.pool.sks-keyservers.net:80 \
> #                          keyserver.ubuntu.com \
> #                          hkp://keyserver.ubuntu.com:80 \
> #                          pgp.mit.edu) ; do \
> #      gpg --batch --keyserver "$server" --recv-keys "$GPG_KEY" && break
> || : ; \
> #  done && \
> #  gpg --batch --verify flink.tgz.asc flink.tgz; \
> #  gpgconf --kill all; \
> #  rm -rf "$GNUPGHOME" flink.tgz.asc; \
> #  \
> #  tar -xf flink.tgz --strip-components=1; \
> #  rm flink.tgz; \
> #  \
>
>
> COPY build-target /opt/flink
> RUN  chown -R flink:flink $FLINK_HOME;
>
> # Configure container
> COPY docker-entrypoint.sh /
> ENTRYPOINT ["/docker-entrypoint.sh"]
> EXPOSE 6123 8081
> CMD ["help"]
> ### END Dockerfile
>
> Am Do., 8. Aug. 2019 um 09:23 Uhr schrieb Kurt Young <[hidden email]>:
>
>> +1 to include this in 1.9.0, adding some examples doesn't look like new
>> feature to me.
>> BTW, I am also trying this tutorial based on release-1.9 branch, but
>> blocked by:
>>
>> git clone --branch release-1.10-SNAPSHOT
>> [hidden email]:apache/flink-playgrounds.git
>>
>> Neither 1.10 nor 1.9 exists in flink-playground yet.
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Aug 8, 2019 at 3:18 PM Fabian Hueske <[hidden email]> wrote:
>>
>> > Hi,
>> > I worked with Konstantin and reviewed the PR.
>> > I think the playground is a great way to get started with Flink and
>> explore
>> > it's recovery mechanism and unique features like savepoints.
>> >
>> > I'm in favor of adding the required streaming example program for the
>> 1.9
>> > release unless there's a good technical argument against it.
>> >
>> > Best, Fabian
>> >
>>
>