[ANNOUNCE] Weekly Community Update 2020/39-43

[Konstantin Knauf-4]

Dear community,

happy to finally share another community update with you. With Flink
Forward and the release of Flink SQL on Ververica Platform I was a bit too
busy the previous weeks. I'll try to return to the weekly cadence going
forward again. This time we have a lot of ground to cover. With the feature
freeze for Flink 1.12 around the corner, this includes some last FLIPs for
this release as well as the first FLIPs that will go into Flink 1.13.

Flink Development
==============

* [releases] It looks like the feature freeze for Flink 1.12 will be set to
Monday, 2nd November. [1]

* [releases] Stateful Function 2.2.0 was released. [2] Please check out the
release blog post [3] for an overview of the new features.

* [releases] Gordon started a discussion on the release of Flink 1.11.3.
Gordon and Xintong will be release managers. Looks like this will happen
soonish. [4]

* [releases] flink-shaded 12.0 was released. [5]

* [sql] Jark has started the discussion on FLIP-145 to support windowing
table-valued functions. This FLIP introduces a new way to do windowing in
Flink SQL, which would eventually supersede the current grouping windows.
It adds four new windowing table-valued functions for TUMBLE, HOP, CUMULATE
and SESSION windows. These functions return a new table with the same
schema as the original table + a "window_start" and "window_end" column.
For every window that a row falls into, one row is added to the output
table. This windowing API provides a better integration with other parts of
Flink SQL and is hopefully easier to pick up for SQL users than the current
syntax. [6]

* [sql] Godfrey has shared a design document to use the multiple-input
stream operator added in FLIP-92 to replace forward shuffles in Flink SQL
by local function calls. [7]

* [sql] [connectors] Jingsong started a discussion and subsequent vote on
FLIP-146. FLIP-146 extends the new TableSource/TableSink interfaces so that
all existing table sources and sinks can be migrated to the new interfaces.
Specifically, it adds the option to create a Table Source/Sink from a
DataStream. [8]

* [sql] [connectors] Shengkai started the discussion and subsequently
the vote on FLIP-149 to support reading/writing Kafka topics in upsert mode
with Flink SQL (aka "Support for Compacted Kafka Topics"). With this FLIP
users will be able to write changelog streams to (presumably compacted)
Kafka topics and Kafka topics can be interpreted as changelog streams
(regular record=upsert, record with null value=delete). [9]

* [deployment] Robert started a discussion on dropping Mesos support in one
of the upcoming releases (earliest Flink 1.13). A lot of support, but also
concerns by current users. So, please if you are Flink on Mesos, let the
community know, so that we can make an informed decision. [10]

* [connectors] Dominik Wosinksi has started a discussion on a MongoDB
(streaming) connector for Apache Flink. There have been tickets/PRs in
Flink and Bahir before and he would like to come to some kind of
conclusion. [11]

* [connectors] Darion Yaphet has asked if it were possible to provide a
connector for the Open Source distributed graph database Nebula Graph. [12]

* [connectors] Kostas proposes to remove the long time deprecated
flink-connector-filesystem. No conclusion yet. [13]

* [runtime] Yingjie Cao proposed to add a sort-merge-based blocking shuffle
to Apache Flink in addition to the hash-style blocking shuffle
implementation that Flink currently uses. The vote is currently ongoing.
[14]

* [docker] Yun Tang started a discussion on using jemalloc as memory
allocator for debian based Flink Docker images. The conclusion seems to be
to make this configurable via an environment variable and to keep glibc
malloc as the default. [15]

* [development process] Aljoscha proposes to use an automatic
code-formatted like spotless going forward. Generally, a lot of support to
use a code-formatted and no strong opinions on the actual code style to be
used. There is some discussion on whether to do this incrementally or all
at once. [16]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Release-1-12-Feature-Freeze-tp45814.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-Stateful-Functions-2-2-0-released-tp45304.html
[3] https://flink.apache.org/news/2020/09/28/release-statefun-2.2.0.html
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Releasing-Apache-Flink-1-11-3-tp45989.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-flink-shaded-12-0-released-tp45453.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-145-Support-SQL-windowing-table-valued-function-tp45269p45270.html
[7]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Multiple-Input-for-Blink-Planner-tp45674.html
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-146-Improve-new-TableSource-and-TableSink-interfaces-tp45161p45172.html
[9]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-149-Introduce-the-KTable-Connector-tp45813.html
[10]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-tp45974.html
[11]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduce-MongoDB-connector-for-Flink-tp45380.html
[12]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Nebula-Connector-tp45885.html
[13]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Remove-flink-connector-filesystem-module-tp45611.html
[14]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-148-Introduce-Sort-Merge-Based-Blocking-Shuffle-to-Flink-tp45717p45727.html
[15]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-docker-Adopt-Jemalloc-as-default-memory-allocator-for-debian-based-Flink-docker-image-tp45643.html
[16]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Enforce-common-opinionated-coding-style-using-Spotless-tp45450p45451.html


Notable Bugs
==========

I've limited my research to the last seven days.

* [FLINK-19790] [1.11.2] Writing MAP column with flink-json produces
incorrect results. Fixed for 1.11.3. [17]
* [FLINK-19711] [1.11.2] [1.10.2] "csv" format does not ignore all parsing
errors even when told so. Fixed for 1.11.3 and 1.10.3. [18]
* [FLINK-19557] [1.11.2] [1.10.2] When a following job manager loses its
connection to Zookeeper and subsequently reconnects, it might miss the fact
that the leader has not changed in the meantime is stuck in "leader
election". Please check the ticket for a more precise description. Fixed
for 1.10.3 and 1.11.3. [19]

[17] https://issues.apache.org/jira/browse/FLINK-19790
[18] https://issues.apache.org/jira/browse/FLINK-19711
[19] https://issues.apache.org/jira/browse/FLINK-19557

Events, Blog Posts, Misc
===================

* Zhu Zhu is now a member of the Apache Flink PMC. [20]

* There are two newish blog post on the Flink blog:

    * Arvid and Stephan published a first part of a blog post series on the
motivation and development of unaligned checkpoints. [21]
<https://flink.apache.org/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html>
   * Gordon published a blog post that looks under the hood of Apache Flink
stateful functions, e.g. explaining the communication protocol between
remote functions and the stateful functions runtime. [22]

* Flink Forward Global took place last week. The recordings of most of the
talks will be available soon.

[20]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-New-PMC-member-Zhu-Zhu-tp45418p45474.html
[21]
https://flink.apache.org/2020/10/15/from-aligned-to-unaligned-checkpoints-part-1.html
[22]
https://flink.apache.org/news/2020/10/13/stateful-serverless-internals.html

Cheers,

Konstantin

--

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk