[ANNOUNCE] Weekly Community Update 2019/43

[Konstantin Knauf-3]

Dear community,

happy to share this week's community update with updates on some exciting
ongoing efforts like unaligned checkpoints, the contribution of a Pulsar
connector and the introduction of Executors, a new initiative around
queryable state, a couple of bugs and a bit more.

Flink Development
==============

* [development process] Gary encourages everyone involved in Flink
development to join the builds@.f.a.o mailing list to receive updates on
build stability issues (e.g. from the nightly builds). You can subscribe
via [hidden email]. [1]

* [state] Piotrs has proposed a change to FLIP-76 on unaligned checkpoints.
For unaligned checkpoints in-flight data needs to be stored as part of the
checkpoint. Piotr proposes to continuously persist all stream records to
avoid having to store a large amount of in-flight data during
checkpointing. Details in the thread [2]

* [state] Vino Yang has revived the discussion on improving queryable state
in Flink. Specifically, he proposes to introduce a new component called
"QueryableStateLocationService": instead of connecting to the
QueryableStateProxy of a Taskmanager a client would connect to the location
service to find the correct TaskManager to serve its query. [3]

* [network] The survey on non credit-based flow control is over and it will
be dropped in the next Flink release. [4]

* [connectors] For the contribution of the Pulsar connector(s), Yijie
kicked off a discussion on a design document for a corresponding FLIP
(FLIP-72). After some discussion, the initial FLIP will now cover mostly
the sink. The source will be part of future work (after FLIP-27) and the
catalog  will get its own FLIP [5].

* [client] Kostas has started (and concluded) a discussion on FLIP-81 to
introduce new configuration options for Executors (FLIP-73). FLIP-81 mainly
introduces configuration options for parameters, that were previously only
exposed via the CLI (e.g. the path to a savepoint for initial state). The
vote is currently ongoing [6,7]

* [sql] Two weeks ago Peter Huang started a discussion on FLIP-59 support
for functions in Flink's DDL. The primary purpose is the dynamic
registration of user defined functions via the DDL. [8]

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/REMINDER-Ensuring-build-stability-td34158.html
[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-76-Unaligned-checkpoints-td33651.html
[3]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Introduce-a-location-oriented-two-stage-query-mechanism-to-improve-the-queryable-state-td34265.html
[4]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Dropping-non-Credit-based-Flow-Control-tp33714.html
[5]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-72-Introduce-Pulsar-Connector-tp33283.html
[6]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/FLIP-81-Executor-related-new-ConfigOptions-tp34236.html
[7]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-81-Executor-related-new-ConfigOptions-tp34290.html
[8]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Discussion-FLIP-79-Flink-Function-DDL-Support-tp33965.html

Notable Bugs
==========

* [FLINK-14074] [1.9.1] [mesos] There is a bug in the way Flink starts
Taskmanagers on Mesos: after the initial set of Taskmanagers is started
(e.g. during the first job submission) any subsequent job submission will
not trigger additional Taskmanagers to be launched although they are needed
to run the job. Check the ticket for a workaround. Fixed for 1.9.2. and
1.10.0.  [9]
* [FLINK-15524[ [1.9.1] The JDBC table sink generates an invalid PostgreSQL
query for upserts. Fixed in 1.9.2 and 1.10.0. [10]
* [FLINK-14470] [1.9.1] [1.8.2] Not particularly new, but still open and
confusing. The Flink WebUI might not show all watermarks for Flink jobs
with many tasks. [11]

[9] https://issues.apache.org/jira/browse/FLINK-14074
[10] https://issues.apache.org/jira/browse/FLINK-14524
[11] https://issues.apache.org/jira/browse/FLINK-14470


Events, Blog Posts, Misc
===================

* Alibaba's food delivery business ele.me has published a blog post about
their stream processing architecture and use cases for Flink. [10]

* There will be Flink/Spark talk at the next Chicago Big Data [11] on the
7th of November. No idea what it will be about (can't join the group) :)

* At the next Athens Big Data Group on the 14th of November *Chaoran Yu *of
Lightbend will talk about Flink and Spark on Kubernetes. [12]

* There will be full-day meetup with six talks in the Bangalore Kafka Group
on the 2nd of November including at least three Flink talks by *Timo Walter*
(Ververica), *Shashank Agarwal* (Razorpay) and *Rasyid Hakim* (GoJek).
[13]

[10]
https://hackernoon.com/flink-or-flunk-why-ele-me-is-developing-a-taste-for-apache-flink-7d2a74e4d6c0
[11]
https://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/events/265675851/
[12] https://www.meetup.com/Athens-Big-Data/events/265957761/
[13] https://www.meetup.com/Bangalore-Apache-Kafka-Group/events/265285812/

Cheers,

Konstantin (@snntrable)

--

Konstantin Knauf | Solutions Architect

[phone]


Follow us @VervericaData Ververica <https://www.ververica.com/>


--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Tony) Cheng