[DISCUSS] Contribute Pulsar Flink connector back to Flink

[Yijie Shen]

Hello everyone,

I've drafted a FLIP that describes the current design of the Pulsar connector:

https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#

Please take a look and let me know what you think.

Thanks,
Yijie

On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]> wrote:

>
> Hi All,
>
> Sorry for joining the discussion late and thanks Yijie & Sijie for driving
> the discussion.
> I also think the Pulsar connector would be a very valuable addition to
> Flink. I can also help out a bit on the review side :-)
>
> Regarding the timeline, I also share concerns with Becket on the
> relationship between the new Pulsar connector and FLIP-27.
> There's also another discussion just started by Stephan on dropping Kafka
> 9/10 support on next Flink release [1].  Although the situation is somewhat
> different, and Kafka 9/10 connector has been in Flink for almost 3-4 years,
> based on the discussion I am not sure if a major version release is a
> requirement for removing old connector supports.
>
> I think there shouldn't be a blocker if we agree the old connector will be
> removed once FLIP-27 based Pulsar connector is there. As Stephan stated, it
> is easier to contribute the source sooner and adjust it later.
> We should also ensure we clearly communicate the message: for example,
> putting an experimental flag on the pre-FLIP27 connector page of the
> website, documentations, etc. Any other thoughts?
>
> --
> Rong
>
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
>
>
> On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]> wrote:
>
> > Technically speaking, removing the old connector code is a backwards
> > incompatible change which requires a major version bump, i.e. Flink 2.x.
> > Given that we don't have a clear plan on when to have the next major
> > version release, it seems unclear how long the old connector code will be
> > there if we check it in right now. Or will we remove the old connector
> > without a major version bump? In any case, it sounds not quite user
> > friendly to the those who might use the old Pulsar connector. I am not sure
> > if it is worth these potential problems in order to have the Pulsar source
> > connector checked in one or two months earlier.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]> wrote:
> >
> > > Agreed, if we check in the old code, we should make it clear that it will
> > > be removed as soon as the FLIP-27 based version of the connector is
> > there.
> > > We should not commit to maintaining the old version, that would be indeed
> > > too much overhead.
> > >
> > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <[hidden email]> wrote:
> > >
> > > > Hi Stephan,
> > > >
> > > > Thanks for the volunteering to help.
> > > >
> > > > Yes, the overhead would just be review capacity. In fact, I am not
> > > worrying
> > > > too much about the review capacity. That is just a one time cost. My
> > > > concern is mainly about the long term burden. Assume we have new source
> > > > interface ready in 1.10 with newly added Pulsar connectors in old
> > > > interface. Later on if we migrate Pulsar to new source interface, the
> > old
> > > > Pulsar connector might be deprecated almost immediately after checked
> > in,
> > > > but we may still have to maintain two code bases. For the existing
> > > > connectors, we have to do that anyways. But it would be good to avoid
> > > > introducing a new connector with the same problem.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <[hidden email]> wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > Nice to see this lively discussion about the Pulsar connector.
> > > > > Some thoughts on the open questions:
> > > > >
> > > > > ## Contribute to Flink or maintain as a community package
> > > > >
> > > > > Looks like the discussion is more going towards contribution. I think
> > > > that
> > > > > is good, especially if we think that we want to build a similarly
> > deep
> > > > > integration with Pulsar as we have for example with Kafka. The
> > > connector
> > > > > already looks like a more thorough connector than many others we have
> > > in
> > > > > the repository.
> > > > >
> > > > > With either a repo split, or the new build system, I hope that the
> > > build
> > > > > overhead is not a problem.
> > > > >
> > > > > ## Committer Support
> > > > >
> > > > > Becket offered some help already, I can also help a bit. I hope that
> > > > > between us, we can cover this.
> > > > >
> > > > > ## Contribute now, or wait for FLIP-27
> > > > >
> > > > > As Becket said, FLIP-27 is actually making some PoC-ing progress, but
> > > > will
> > > > > take 2 more months, I would estimate, before it is fully available.
> > > > >
> > > > > If we want to be on the safe side with the contribution, we should
> > > > > contribute the source sooner and adjust it later. That would also
> > help
> > > us
> > > > > in case things get crazy towards the 1.10 feature freeze and it would
> > > be
> > > > > hard to find time to review the new changes.
> > > > > What would be the overhead of contributing now? Given that the code
> > is
> > > > > already there, it looks like it would be only review capacity, right?
> > > > >
> > > > > Best,
> > > > > Stephan
> > > > >
> > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi everyone!
> > > > > >
> > > > > > Thanks for your attention and the promotion of this work.
> > > > > >
> > > > > > We will prepare a FLIP as soon as possible for more specific
> > > > discussions.
> > > > > >
> > > > > > For FLIP-27, it seems that we have not reached a consensus.
> > > Therefore,
> > > > > > I will explain all the functionalities of the existing connector in
> > > > > > the FLIP (including Source, Sink, and Catalog) to continue our
> > > > > > discussions in FLIP.
> > > > > >
> > > > > > Thanks for your kind help.
> > > > > >
> > > > > > Best,
> > > > > > Yijie
> > > > > >
> > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <[hidden email]>
> > > > wrote:
> > > > > > >
> > > > > > > Hi Sijie,
> > > > > > >
> > > > > > > If we agree that the goal is to have Pulsar connector in 1.10,
> > how
> > > > > about
> > > > > > we
> > > > > > > do the following:
> > > > > > >
> > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main repo as it
> > > is a
> > > > > new
> > > > > > > public interface to Flink main repo.
> > > > > > > 1. Start to review the Pulsar sink right away as there is no
> > change
> > > > to
> > > > > > the
> > > > > > > sink interface so far.
> > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to be code
> > > > freeze
> > > > > in
> > > > > > > late Nov and let's say we give a month to the development and
> > > review
> > > > of
> > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct. There are
> > > > still
> > > > > 7
> > > > > > > weeks. Personally I think it is doable. If FLIP-27 is not ready
> > by
> > > > late
> > > > > > > Oct, we can review and check in Pulsar connector with the
> > existing
> > > > > source
> > > > > > > interface. This means we will have Pulsar connector in Flink
> > 1.10,
> > > > > either
> > > > > > > with or without FLIP-27.
> > > > > > >
> > > > > > > Because we are going to have Pulsar sink and source checked in
> > > > > > separately,
> > > > > > > it might make sense to have two FLIPs, one for Pulsar sink and
> > > > another
> > > > > > for
> > > > > > > Pulsar source. And we can start the work on Pulsar sink right
> > away.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <[hidden email]>
> > > wrote:
> > > > > > >
> > > > > > > > Thank you Bowen and Becket.
> > > > > > > >
> > > > > > > > What's the take from Flink community? Shall we wait for FLIP-27
> > > or
> > > > > > shall we
> > > > > > > > proceed to next steps? And what the next steps are? :-)
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Sijie
> > > > > > > >
> > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <[hidden email]>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I think having a Pulsar connector in Flink can be a good
> > mutual
> > > > > > benefit
> > > > > > > > to
> > > > > > > > > both communities.
> > > > > > > > >
> > > > > > > > > Another perspective is that Pulsar connector is the 1st
> > > streaming
> > > > > > > > connector
> > > > > > > > > that integrates with Flink's metadata management system and
> > > > Catalog
> > > > > > APIs.
> > > > > > > > > It'll be cool to see how the integration turns out and
> > whether
> > > we
> > > > > > need to
> > > > > > > > > improve Flink Catalog stack, which are currently in Beta, to
> > > > cater
> > > > > to
> > > > > > > > > streaming source/sink. Thus I'm in favor of merging Pulsar
> > > > > connector
> > > > > > into
> > > > > > > > > Flink 1.10.
> > > > > > > > >
> > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe one for
> > > basic
> > > > > > > > > source/sink functionalities and another for schema and
> > catalog
> > > > > > > > integration,
> > > > > > > > > just to make them easier to review.
> > > > > > > > >
> > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I don't
> > think
> > > > > > FLIP-27
> > > > > > > > > should be a blocker in cases where it cannot make its way
> > into
> > > > 1.10
> > > > > > or
> > > > > > > > > doesn't leave reasonable amount of time for committers to
> > > review
> > > > or
> > > > > > for
> > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > >
> > > > > > > > > Bowen
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > [hidden email]>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Till,
> > > > > > > > > >
> > > > > > > > > > You are right. It all depends on when the new source
> > > interface
> > > > is
> > > > > > going
> > > > > > > > > to
> > > > > > > > > > be ready. Personally I think it would be there in about a
> > > month
> > > > > or
> > > > > > so.
> > > > > > > > > But
> > > > > > > > > > I could be too optimistic. It would also be good to hear
> > what
> > > > do
> > > > > > > > Aljoscha
> > > > > > > > > > and Stephan think as they are also involved in FLIP-27.
> > > > > > > > > >
> > > > > > > > > > In general I think we should have Pulsar connector in Flink
> > > > 1.10,
> > > > > > > > > > preferably with the new source interface. We can also check
> > > it
> > > > in
> > > > > > right
> > > > > > > > > now
> > > > > > > > > > with old source interface, but I suspect few users will use
> > > it
> > > > > > before
> > > > > > > > the
> > > > > > > > > > next official release. Therefore, it seems reasonable to
> > > wait a
> > > > > > little
> > > > > > > > > bit
> > > > > > > > > > to see whether we can jump to the new source interface. As
> > > long
> > > > > as
> > > > > > we
> > > > > > > > > make
> > > > > > > > > > sure Flink 1.10 has it, waiting a little bit doesn't seem
> > to
> > > > hurt
> > > > > > much.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > I'm wondering what the problem would be if we committed
> > the
> > > > > > Pulsar
> > > > > > > > > > > connector before the new source interface is ready. If I
> > > > > > understood
> > > > > > > > it
> > > > > > > > > > > correctly, then we need to support the old source
> > interface
> > > > > > anyway
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > existing connectors. By checking it in early I could see
> > > the
> > > > > > benefit
> > > > > > > > > that
> > > > > > > > > > > our users could start using the connector earlier.
> > > Moreover,
> > > > it
> > > > > > would
> > > > > > > > > > > prevent that the Pulsar integration is being delayed in
> > > case
> > > > > > that the
> > > > > > > > > > > source interface should be delayed. The only downside I
> > see
> > > > is
> > > > > > the
> > > > > > > > > extra
> > > > > > > > > > > review effort and potential fixes which might be
> > irrelevant
> > > > for
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > source interface implementation. I guess it mainly
> > depends
> > > on
> > > > > how
> > > > > > > > > certain
> > > > > > > > > > > we are when the new source interface will be ready.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Till
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > >
> > > > > > > > > > > > Just want to have some update on FLIP-27. Although the
> > > FLIP
> > > > > > wiki
> > > > > > > > and
> > > > > > > > > > > > discussion thread has been quiet for some time, a few
> > > > > > committer /
> > > > > > > > > > > > contributors in Flink community were actually
> > prototyping
> > > > the
> > > > > > > > entire
> > > > > > > > > > > thing.
> > > > > > > > > > > > We have made some good progress there but want to
> > update
> > > > the
> > > > > > FLIP
> > > > > > > > > wiki
> > > > > > > > > > > > after the entire thing is verified to work in case
> > there
> > > > are
> > > > > > some
> > > > > > > > > last
> > > > > > > > > > > > minute surprise in the implementation. I don't have an
> > > > exact
> > > > > > ETA
> > > > > > > > yet,
> > > > > > > > > > > but I
> > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > >
> > > > > > > > > > > > I am happy to review the current Flink Pulsar connector
> > > and
> > > > > > see if
> > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the case that
> > > we
> > > > > > checked
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > Pulsar connector with some review efforts and shortly
> > > after
> > > > > > that
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for all the feedback and suggestions!
> > > > > > > > > > > > >
> > > > > > > > > > > > > As Sijie said, the goal of the connector has always
> > > been
> > > > to
> > > > > > > > provide
> > > > > > > > > > > > > users with the latest features of both systems as
> > soon
> > > as
> > > > > > > > possible.
> > > > > > > > > > We
> > > > > > > > > > > > > propose to contribute the connector to Flink and hope
> > > to
> > > > > get
> > > > > > more
> > > > > > > > > > > > > suggestions and feedback from Flink experts to ensure
> > > the
> > > > > > high
> > > > > > > > > > quality
> > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For FLIP-27, we noticed its existence at the
> > beginning
> > > of
> > > > > > > > reworking
> > > > > > > > > > > > > the connector implementation based on Flink 1.9; we
> > > also
> > > > > > wanted
> > > > > > > > to
> > > > > > > > > > > > > build a connector that supports both batch and stream
> > > > > > computing
> > > > > > > > > based
> > > > > > > > > > > > > on it.
> > > > > > > > > > > > > However, it has been inactive for some time, so we
> > > > decided
> > > > > to
> > > > > > > > > provide
> > > > > > > > > > > > > a connector with most of the new features, such as
> > the
> > > > new
> > > > > > type
> > > > > > > > > > system
> > > > > > > > > > > > > and the new catalog API first. We will pay attention
> > to
> > > > the
> > > > > > > > > progress
> > > > > > > > > > > > > of FLIP-27 continually and incorporate it with the
> > > > > connector
> > > > > > as
> > > > > > > > > soon
> > > > > > > > > > > > > as possible.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding the test status of the connector, we are
> > > > > following
> > > > > > the
> > > > > > > > > > other
> > > > > > > > > > > > > connectors' test in Flink repository and aimed to
> > > provide
> > > > > > > > > throughout
> > > > > > > > > > > > > tests as we could. We are also happy to hear
> > > suggestions
> > > > > and
> > > > > > > > > > > > > supervision from the Flink community to improve the
> > > > > > stability and
> > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Yijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks everyone for the comments and feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It seems to me that the main question here is
> > about -
> > > > > "how
> > > > > > can
> > > > > > > > > the
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1) I think how and where to host this integration
> > is
> > > > kind
> > > > > > of
> > > > > > > > less
> > > > > > > > > > > > > important
> > > > > > > > > > > > > > here. I believe there can be many ways to achieve
> > it.
> > > > > > > > > > > > > > As part of the contribution, what we are looking
> > for
> > > > here
> > > > > > is
> > > > > > > > how
> > > > > > > > > > > these
> > > > > > > > > > > > > two
> > > > > > > > > > > > > > communities can build the collaboration
> > relationship
> > > on
> > > > > > > > > developing
> > > > > > > > > > > > > > the integration between Pulsar and Flink. Even we
> > can
> > > > try
> > > > > > our
> > > > > > > > > best
> > > > > > > > > > to
> > > > > > > > > > > > > catch
> > > > > > > > > > > > > > up all the updates in Flink community. We are still
> > > > > > > > > > > > > > facing the fact that we have less experiences in
> > > Flink
> > > > > than
> > > > > > > > folks
> > > > > > > > > > in
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > community. In order to make sure we maintain and
> > > > deliver
> > > > > > > > > > > > > > a high-quality pulsar-flink integration to the
> > users
> > > > who
> > > > > > use
> > > > > > > > both
> > > > > > > > > > > > > > technologies, we need some help from the experts
> > from
> > > > > Flink
> > > > > > > > > > > community.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2) We have been following FLIP-27 for a while.
> > > > Originally
> > > > > > we
> > > > > > > > were
> > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > of contributing the connectors back after
> > integrating
> > > > > with
> > > > > > the
> > > > > > > > > > > > > > new API introduced in FLIP-27. But we decided to
> > > > initiate
> > > > > > the
> > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > as early as possible. Because we believe there are
> > > more
> > > > > > > > benefits
> > > > > > > > > > > doing
> > > > > > > > > > > > > > it now rather than later. As part of contribution,
> > it
> > > > can
> > > > > > help
> > > > > > > > > > Flink
> > > > > > > > > > > > > > community understand more about Pulsar and the
> > > > potential
> > > > > > > > > > integration
> > > > > > > > > > > > > points.
> > > > > > > > > > > > > > Also we can also help Flink community verify the
> > new
> > > > > > connector
> > > > > > > > > API
> > > > > > > > > > as
> > > > > > > > > > > > > well
> > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket Qin <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the interest in contributing the
> > Pulsar
> > > > > > connector.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In general, I think having Pulsar connector with
> > > > strong
> > > > > > > > support
> > > > > > > > > > is
> > > > > > > > > > > a
> > > > > > > > > > > > > > > valuable addition to Flink. So I am happy the
> > > > shepherd
> > > > > > this
> > > > > > > > > > effort.
> > > > > > > > > > > > > > > Meanwhile, I would also like to provide some
> > > context
> > > > > and
> > > > > > > > recent
> > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The current way Flink maintains its connector has
> > > hit
> > > > > the
> > > > > > > > > > > scalability
> > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > With more and more connectors coming into Flink
> > > repo,
> > > > > we
> > > > > > are
> > > > > > > > > > > facing a
> > > > > > > > > > > > > few
> > > > > > > > > > > > > > > problems such as long build and testing time. To
> > > > > address
> > > > > > this
> > > > > > > > > > > > problem,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > 1. Split out the connectors into a separate
> > > > repository.
> > > > > > This
> > > > > > > > is
> > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > on hold due to potential solution to shorten the
> > > > build
> > > > > > time.
> > > > > > > > > > > > > > > 2. Encourage the connectors to stay as ecosystem
> > > > > project
> > > > > > > > while
> > > > > > > > > > > Flink
> > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > to provide good support for functionality and
> > > > > > compatibility
> > > > > > > > > > tests.
> > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > has driven to create a Flink Ecosystem project
> > > > website
> > > > > > and it
> > > > > > > > > is
> > > > > > > > > > > > going
> > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Given the above efforts, it would be great to
> > first
> > > > see
> > > > > > if we
> > > > > > > > > can
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > Pulsar connector as an ecosystem project with
> > great
> > > > > > support.
> > > > > > > > It
> > > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > good to hear how the Flink Pulsar connector is
> > > tested
> > > > > > > > currently
> > > > > > > > > > to
> > > > > > > > > > > > see
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > we can learn something to maintain it as an
> > > ecosystem
> > > > > > project
> > > > > > > > > > with
> > > > > > > > > > > > good
> > > > > > > > > > > > > > > quality and test coverage. If the quality as an
> > > > > ecosystem
> > > > > > > > > project
> > > > > > > > > > > is
> > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > to guarantee, we may as well adopt it into the
> > main
> > > > > repo.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27 where we
> > are
> > > > > > making
> > > > > > > > > > changes
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > Flink source connector architecture and
> > interface.
> > > > This
> > > > > > > > change
> > > > > > > > > > will
> > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if we are
> > > going
> > > > to
> > > > > > have
> > > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > connector in main repo, I am wondering if we
> > should
> > > > > hold
> > > > > > a
> > > > > > > > > little
> > > > > > > > > > > bit
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > let the Pulsar connector adapt to the new
> > interface
> > > > to
> > > > > > avoid
> > > > > > > > > > > shortly
> > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay Schepler <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm quite worried that we may end up repeating
> > > > > history.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > There were already 2 attempts at contributing a
> > > > > pulsar
> > > > > > > > > > connector,
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > of which failed because no committer was
> > getting
> > > > > > involved,
> > > > > > > > > > > despite
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > contributor opening a dedicated discussion
> > thread
> > > > > > about the
> > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > beforehand and getting several +1's from
> > > > committers.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We should really make sure that if we
> > > > welcome/approve
> > > > > > such
> > > > > > > > a
> > > > > > > > > > > > > > > > contribution it will actually get the attention
> > > it
> > > > > > > > deserves.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As such, I'm inclined to recommend maintaining
> > > the
> > > > > > > > connector
> > > > > > > > > > > > outside
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Flink. We could link to it from the
> > documentation
> > > > to
> > > > > > give
> > > > > > > > it
> > > > > > > > > > more
> > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > With the upcoming page for sharing artifacts
> > > among
> > > > > the
> > > > > > > > > > community
> > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > the state of that anyway?), this may be a
> > better
> > > > > > option.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann wrote:
> > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > thanks a lot for starting this discussion
> > > Yijie.
> > > > I
> > > > > > think
> > > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > connector would be a very valuable addition
> > > since
> > > > > > Pulsar
> > > > > > > > > > > becomes
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > more popular and it would further expand
> > > Flink's
> > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > from a project perspective it makes sense for
> > > me
> > > > to
> > > > > > place
> > > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > My main concern/question is how can the Flink
> > > > > > community
> > > > > > > > > > > maintain
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector? We have seen in the past that
> > > > connectors
> > > > > > are
> > > > > > > > > some
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > actively developed components because they
> > need
> > > > to
> > > > > be
> > > > > > > > kept
> > > > > > > > > in
> > > > > > > > > > > > sync
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > external system and with Flink. Given that
> > the
> > > > > Pulsar
> > > > > > > > > > community
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > to help with maintaining, improving and
> > > evolving
> > > > > the
> > > > > > > > > > connector,
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > optimistic that we can achieve this. Hence,
> > +1
> > > > for
> > > > > > > > > > contributing
> > > > > > > > > > > > it
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM Sijie Guo <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > FLINK-9641
> > > > and
> > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > let
> > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was created for
> > > > bringing
> > > > > > > > Pulsar
> > > > > > > > > as
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> sink for Flink. The integration was done
> > with
> > > > > Flink
> > > > > > > > 1.6.0.
> > > > > > > > > > We
> > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > >> requests about a year ago and we ended up
> > > > > > maintaining
> > > > > > > > > those
> > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use Flink to
> > > process
> > > > > > event
> > > > > > > > > > streams
> > > > > > > > > > > in
> > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > >
> > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > ).
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple and there
> > is
> > > no
> > > > > > schema
> > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> In the past year, we have made a lot of
> > > changes
> > > > in
> > > > > > > > Pulsar
> > > > > > > > > > and
> > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > >> Pulsar schema as the first-class citizen in
> > > > > Pulsar.
> > > > > > We
> > > > > > > > > also
> > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > >> other computing engines for processing
> > Pulsar
> > > > > event
> > > > > > > > > streams
> > > > > > > > > > > with
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> It led us to rethink how to integrate with
> > > Flink
> > > > > in
> > > > > > the
> > > > > > > > > best
> > > > > > > > > > > > way.
> > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > >> reimplement the pulsar-flink connectors from
> > > the
> > > > > > ground
> > > > > > > > up
> > > > > > > > > > > with
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> bring table API and catalog API as the
> > > > first-class
> > > > > > > > citizen
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >> integration. With that being said, in the
> > new
> > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > >> you can register pulsar as a flink catalog
> > and
> > > > > > query /
> > > > > > > > > > process
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> This is an example about how to use Pulsar
> > as
> > > a
> > > > > > Flink
> > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > explaining
> > > > why
> > > > > we
> > > > > > > > > > > > re-implement
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what are the
> > > > changes
> > > > > we
> > > > > > > > made
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> We believe Pulsar is not just a simple data
> > > sink
> > > > > or
> > > > > > > > source
> > > > > > > > > > for
> > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > >> actually can be a fully integrated streaming
> > > > data
> > > > > > > > storage
> > > > > > > > > > for
> > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > >> areas (sink, source, schema/catalog and
> > > state).
> > > > > The
> > > > > > > > > > > combination
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > >> and Pulsar can create a great streaming
> > > > warehouse
> > > > > > > > > > architecture
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > >> streaming-first, unified data processing.
> > > Since
> > > > we
> > > > > > are
> > > > > > > > > > talking
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > >> contribute Pulsar integration to Flink here,
> > > we
> > > > > are
> > > > > > also
> > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > >> maintain, improve and evolve the integration
> > > > with
> > > > > > Flink
> > > > > > > > to
> > > > > > > > > > > help
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Hope this give you a bit more background
> > about
> > > > the
> > > > > > > > pulsar
> > > > > > > > > > > flink
> > > > > > > > > > > > > > > > >> integration. Let me know what are your
> > > thoughts.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM Yun Tang <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes more and more
> > > > > popular
> > > > > > > > > > recently
> > > > > > > > > > > > and
> > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > >>> to see more people willing to contribute to
> > > > Flink
> > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Before any further discussion, would you
> > > please
> > > > > > give
> > > > > > > > some
> > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >>> the relationship between this thread to
> > > current
> > > > > > > > existing
> > > > > > > > > > > JIRAs
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > >>> source [1] and sink [2] connector? Will the
> > > > > > > > contribution
> > > > > > > > > > > > contains
> > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > implementation?
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> [1]
> > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > >>> [2]
> > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > [hidden email]>
> > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019 13:57
> > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > [hidden email]
> > > > >
> > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute Pulsar Flink
> > > > > > connector
> > > > > > > > back
> > > > > > > > > > to
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> I would like to open the discussion of
> > > > > contributing
> > > > > > > > > Pulsar
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache Pulsar
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a multi-tenant,
> > > > > > high-performance
> > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar includes
> > > > > multiple
> > > > > > > > > features
> > > > > > > > > > > > such
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > >>> native support for multiple clusters in a
> > > > Pulsar
> > > > > > > > > instance,
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > >>> seamless geo-replication of messages across
> > > > > > clusters,
> > > > > > > > > very
> > > > > > > > > > > low
> > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > scalability
> > > to
> > > > > > over a
> > > > > > > > > > > million
> > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > >>> and guaranteed message delivery with
> > > persistent
> > > > > > message
> > > > > > > > > > > storage
> > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper. Nowadays,
> > > Pulsar
> > > > > has
> > > > > > > > been
> > > > > > > > > > > > adopted
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink connector
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are planning
> > to
> > > > > > > > contribute
> > > > > > > > > is
> > > > > > > > > > > > built
> > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The main
> > > features
> > > > > > are:
> > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source with
> > > > exactly-once
> > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > >>> - Sink streaming results to Pulsar with
> > > > > > at-least-once
> > > > > > > > > > > > semantics.
> > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > >>> would update this to exactly-once as well
> > > when
> > > > > > Pulsar
> > > > > > > > > gets
> > > > > > > > > > > all
> > > > > > > > > > > > > > > > >>> transaction features ready in its 2.5.0
> > > > version)
> > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API Type
> > system
> > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > and
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > >>> automatically (de)serialize messages with
> > the
> > > > > help
> > > > > > of
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > >>> - Integrate with Flink new Catalog API
> > > > > > (FLIP-30[4]),
> > > > > > > > > which
> > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in Table API
> > > as
> > > > > > well as
> > > > > > > > > SQL
> > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > >>> [0]
> > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > >>> [2]
> > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

[Rong Rong]

Hi Yijie,

Thanks for sharing the pulsar FLIP.
Would you mind enabling comments/suggestions on the google doc link? This
way the contributors from the community can comment on the doc.

Best,
Rong

On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <[hidden email]>
wrote:

> Hello everyone,
>
> I've drafted a FLIP that describes the current design of the Pulsar
> connector:
>
>
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
>
> Please take a look and let me know what you think.
>
> Thanks,
> Yijie
>
> On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]> wrote:
> >
> > Hi All,
> >
> > Sorry for joining the discussion late and thanks Yijie & Sijie for
> driving
> > the discussion.
> > I also think the Pulsar connector would be a very valuable addition to
> > Flink. I can also help out a bit on the review side :-)
> >
> > Regarding the timeline, I also share concerns with Becket on the
> > relationship between the new Pulsar connector and FLIP-27.
> > There's also another discussion just started by Stephan on dropping Kafka
> > 9/10 support on next Flink release [1].  Although the situation is
> somewhat
> > different, and Kafka 9/10 connector has been in Flink for almost 3-4
> years,
> > based on the discussion I am not sure if a major version release is a
> > requirement for removing old connector supports.
> >
> > I think there shouldn't be a blocker if we agree the old connector will
> be
> > removed once FLIP-27 based Pulsar connector is there. As Stephan stated,
> it
> > is easier to contribute the source sooner and adjust it later.
> > We should also ensure we clearly communicate the message: for example,
> > putting an experimental flag on the pre-FLIP27 connector page of the
> > website, documentations, etc. Any other thoughts?
> >
> > --
> > Rong
> >
> > [1]
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> >
> >
> > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]> wrote:
> >
> > > Technically speaking, removing the old connector code is a backwards
> > > incompatible change which requires a major version bump, i.e. Flink
> 2.x.
> > > Given that we don't have a clear plan on when to have the next major
> > > version release, it seems unclear how long the old connector code will
> be
> > > there if we check it in right now. Or will we remove the old connector
> > > without a major version bump? In any case, it sounds not quite user
> > > friendly to the those who might use the old Pulsar connector. I am not
> sure
> > > if it is worth these potential problems in order to have the Pulsar
> source
> > > connector checked in one or two months earlier.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]> wrote:
> > >
> > > > Agreed, if we check in the old code, we should make it clear that it
> will
> > > > be removed as soon as the FLIP-27 based version of the connector is
> > > there.
> > > > We should not commit to maintaining the old version, that would be
> indeed
> > > > too much overhead.
> > > >
> > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <[hidden email]>
> wrote:
> > > >
> > > > > Hi Stephan,
> > > > >
> > > > > Thanks for the volunteering to help.
> > > > >
> > > > > Yes, the overhead would just be review capacity. In fact, I am not
> > > > worrying
> > > > > too much about the review capacity. That is just a one time cost.
> My
> > > > > concern is mainly about the long term burden. Assume we have new
> source
> > > > > interface ready in 1.10 with newly added Pulsar connectors in old
> > > > > interface. Later on if we migrate Pulsar to new source interface,
> the
> > > old
> > > > > Pulsar connector might be deprecated almost immediately after
> checked
> > > in,
> > > > > but we may still have to maintain two code bases. For the existing
> > > > > connectors, we have to do that anyways. But it would be good to
> avoid
> > > > > introducing a new connector with the same problem.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <[hidden email]>
> wrote:
> > > > >
> > > > > > Hi all!
> > > > > >
> > > > > > Nice to see this lively discussion about the Pulsar connector.
> > > > > > Some thoughts on the open questions:
> > > > > >
> > > > > > ## Contribute to Flink or maintain as a community package
> > > > > >
> > > > > > Looks like the discussion is more going towards contribution. I
> think
> > > > > that
> > > > > > is good, especially if we think that we want to build a similarly
> > > deep
> > > > > > integration with Pulsar as we have for example with Kafka. The
> > > > connector
> > > > > > already looks like a more thorough connector than many others we
> have
> > > > in
> > > > > > the repository.
> > > > > >
> > > > > > With either a repo split, or the new build system, I hope that
> the
> > > > build
> > > > > > overhead is not a problem.
> > > > > >
> > > > > > ## Committer Support
> > > > > >
> > > > > > Becket offered some help already, I can also help a bit. I hope
> that
> > > > > > between us, we can cover this.
> > > > > >
> > > > > > ## Contribute now, or wait for FLIP-27
> > > > > >
> > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> progress, but
> > > > > will
> > > > > > take 2 more months, I would estimate, before it is fully
> available.
> > > > > >
> > > > > > If we want to be on the safe side with the contribution, we
> should
> > > > > > contribute the source sooner and adjust it later. That would also
> > > help
> > > > us
> > > > > > in case things get crazy towards the 1.10 feature freeze and it
> would
> > > > be
> > > > > > hard to find time to review the new changes.
> > > > > > What would be the overhead of contributing now? Given that the
> code
> > > is
> > > > > > already there, it looks like it would be only review capacity,
> right?
> > > > > >
> > > > > > Best,
> > > > > > Stephan
> > > > > >
> > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone!
> > > > > > >
> > > > > > > Thanks for your attention and the promotion of this work.
> > > > > > >
> > > > > > > We will prepare a FLIP as soon as possible for more specific
> > > > > discussions.
> > > > > > >
> > > > > > > For FLIP-27, it seems that we have not reached a consensus.
> > > > Therefore,
> > > > > > > I will explain all the functionalities of the existing
> connector in
> > > > > > > the FLIP (including Source, Sink, and Catalog) to continue our
> > > > > > > discussions in FLIP.
> > > > > > >
> > > > > > > Thanks for your kind help.
> > > > > > >
> > > > > > > Best,
> > > > > > > Yijie
> > > > > > >
> > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> [hidden email]>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Sijie,
> > > > > > > >
> > > > > > > > If we agree that the goal is to have Pulsar connector in
> 1.10,
> > > how
> > > > > > about
> > > > > > > we
> > > > > > > > do the following:
> > > > > > > >
> > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main repo
> as it
> > > > is a
> > > > > > new
> > > > > > > > public interface to Flink main repo.
> > > > > > > > 1. Start to review the Pulsar sink right away as there is no
> > > change
> > > > > to
> > > > > > > the
> > > > > > > > sink interface so far.
> > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to be
> code
> > > > > freeze
> > > > > > in
> > > > > > > > late Nov and let's say we give a month to the development and
> > > > review
> > > > > of
> > > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct. There
> are
> > > > > still
> > > > > > 7
> > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is not
> ready
> > > by
> > > > > late
> > > > > > > > Oct, we can review and check in Pulsar connector with the
> > > existing
> > > > > > source
> > > > > > > > interface. This means we will have Pulsar connector in Flink
> > > 1.10,
> > > > > > either
> > > > > > > > with or without FLIP-27.
> > > > > > > >
> > > > > > > > Because we are going to have Pulsar sink and source checked
> in
> > > > > > > separately,
> > > > > > > > it might make sense to have two FLIPs, one for Pulsar sink
> and
> > > > > another
> > > > > > > for
> > > > > > > > Pulsar source. And we can start the work on Pulsar sink right
> > > away.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <[hidden email]
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > >
> > > > > > > > > What's the take from Flink community? Shall we wait for
> FLIP-27
> > > > or
> > > > > > > shall we
> > > > > > > > > proceed to next steps? And what the next steps are? :-)
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Sijie
> > > > > > > > >
> > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> [hidden email]>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I think having a Pulsar connector in Flink can be a good
> > > mutual
> > > > > > > benefit
> > > > > > > > > to
> > > > > > > > > > both communities.
> > > > > > > > > >
> > > > > > > > > > Another perspective is that Pulsar connector is the 1st
> > > > streaming
> > > > > > > > > connector
> > > > > > > > > > that integrates with Flink's metadata management system
> and
> > > > > Catalog
> > > > > > > APIs.
> > > > > > > > > > It'll be cool to see how the integration turns out and
> > > whether
> > > > we
> > > > > > > need to
> > > > > > > > > > improve Flink Catalog stack, which are currently in
> Beta, to
> > > > > cater
> > > > > > to
> > > > > > > > > > streaming source/sink. Thus I'm in favor of merging
> Pulsar
> > > > > > connector
> > > > > > > into
> > > > > > > > > > Flink 1.10.
> > > > > > > > > >
> > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe one
> for
> > > > basic
> > > > > > > > > > source/sink functionalities and another for schema and
> > > catalog
> > > > > > > > > integration,
> > > > > > > > > > just to make them easier to review.
> > > > > > > > > >
> > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I don't
> > > think
> > > > > > > FLIP-27
> > > > > > > > > > should be a blocker in cases where it cannot make its way
> > > into
> > > > > 1.10
> > > > > > > or
> > > > > > > > > > doesn't leave reasonable amount of time for committers to
> > > > review
> > > > > or
> > > > > > > for
> > > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > > >
> > > > > > > > > > Bowen
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Till,
> > > > > > > > > > >
> > > > > > > > > > > You are right. It all depends on when the new source
> > > > interface
> > > > > is
> > > > > > > going
> > > > > > > > > > to
> > > > > > > > > > > be ready. Personally I think it would be there in
> about a
> > > > month
> > > > > > or
> > > > > > > so.
> > > > > > > > > > But
> > > > > > > > > > > I could be too optimistic. It would also be good to
> hear
> > > what
> > > > > do
> > > > > > > > > Aljoscha
> > > > > > > > > > > and Stephan think as they are also involved in FLIP-27.
> > > > > > > > > > >
> > > > > > > > > > > In general I think we should have Pulsar connector in
> Flink
> > > > > 1.10,
> > > > > > > > > > > preferably with the new source interface. We can also
> check
> > > > it
> > > > > in
> > > > > > > right
> > > > > > > > > > now
> > > > > > > > > > > with old source interface, but I suspect few users
> will use
> > > > it
> > > > > > > before
> > > > > > > > > the
> > > > > > > > > > > next official release. Therefore, it seems reasonable
> to
> > > > wait a
> > > > > > > little
> > > > > > > > > > bit
> > > > > > > > > > > to see whether we can jump to the new source
> interface. As
> > > > long
> > > > > > as
> > > > > > > we
> > > > > > > > > > make
> > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit doesn't
> seem
> > > to
> > > > > hurt
> > > > > > > much.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm wondering what the problem would be if we
> committed
> > > the
> > > > > > > Pulsar
> > > > > > > > > > > > connector before the new source interface is ready.
> If I
> > > > > > > understood
> > > > > > > > > it
> > > > > > > > > > > > correctly, then we need to support the old source
> > > interface
> > > > > > > anyway
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > existing connectors. By checking it in early I could
> see
> > > > the
> > > > > > > benefit
> > > > > > > > > > that
> > > > > > > > > > > > our users could start using the connector earlier.
> > > > Moreover,
> > > > > it
> > > > > > > would
> > > > > > > > > > > > prevent that the Pulsar integration is being delayed
> in
> > > > case
> > > > > > > that the
> > > > > > > > > > > > source interface should be delayed. The only
> downside I
> > > see
> > > > > is
> > > > > > > the
> > > > > > > > > > extra
> > > > > > > > > > > > review effort and potential fixes which might be
> > > irrelevant
> > > > > for
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > source interface implementation. I guess it mainly
> > > depends
> > > > on
> > > > > > how
> > > > > > > > > > certain
> > > > > > > > > > > > we are when the new source interface will be ready.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Till
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Just want to have some update on FLIP-27. Although
> the
> > > > FLIP
> > > > > > > wiki
> > > > > > > > > and
> > > > > > > > > > > > > discussion thread has been quiet for some time, a
> few
> > > > > > > committer /
> > > > > > > > > > > > > contributors in Flink community were actually
> > > prototyping
> > > > > the
> > > > > > > > > entire
> > > > > > > > > > > > thing.
> > > > > > > > > > > > > We have made some good progress there but want to
> > > update
> > > > > the
> > > > > > > FLIP
> > > > > > > > > > wiki
> > > > > > > > > > > > > after the entire thing is verified to work in case
> > > there
> > > > > are
> > > > > > > some
> > > > > > > > > > last
> > > > > > > > > > > > > minute surprise in the implementation. I don't
> have an
> > > > > exact
> > > > > > > ETA
> > > > > > > > > yet,
> > > > > > > > > > > > but I
> > > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am happy to review the current Flink Pulsar
> connector
> > > > and
> > > > > > > see if
> > > > > > > > > it
> > > > > > > > > > > > would
> > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the case
> that
> > > > we
> > > > > > > checked
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > Pulsar connector with some review efforts and
> shortly
> > > > after
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for all the feedback and suggestions!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As Sijie said, the goal of the connector has
> always
> > > > been
> > > > > to
> > > > > > > > > provide
> > > > > > > > > > > > > > users with the latest features of both systems as
> > > soon
> > > > as
> > > > > > > > > possible.
> > > > > > > > > > > We
> > > > > > > > > > > > > > propose to contribute the connector to Flink and
> hope
> > > > to
> > > > > > get
> > > > > > > more
> > > > > > > > > > > > > > suggestions and feedback from Flink experts to
> ensure
> > > > the
> > > > > > > high
> > > > > > > > > > > quality
> > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For FLIP-27, we noticed its existence at the
> > > beginning
> > > > of
> > > > > > > > > reworking
> > > > > > > > > > > > > > the connector implementation based on Flink 1.9;
> we
> > > > also
> > > > > > > wanted
> > > > > > > > > to
> > > > > > > > > > > > > > build a connector that supports both batch and
> stream
> > > > > > > computing
> > > > > > > > > > based
> > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > However, it has been inactive for some time, so
> we
> > > > > decided
> > > > > > to
> > > > > > > > > > provide
> > > > > > > > > > > > > > a connector with most of the new features, such
> as
> > > the
> > > > > new
> > > > > > > type
> > > > > > > > > > > system
> > > > > > > > > > > > > > and the new catalog API first. We will pay
> attention
> > > to
> > > > > the
> > > > > > > > > > progress
> > > > > > > > > > > > > > of FLIP-27 continually and incorporate it with
> the
> > > > > > connector
> > > > > > > as
> > > > > > > > > > soon
> > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the test status of the connector, we
> are
> > > > > > following
> > > > > > > the
> > > > > > > > > > > other
> > > > > > > > > > > > > > connectors' test in Flink repository and aimed to
> > > > provide
> > > > > > > > > > throughout
> > > > > > > > > > > > > > tests as we could. We are also happy to hear
> > > > suggestions
> > > > > > and
> > > > > > > > > > > > > > supervision from the Flink community to improve
> the
> > > > > > > stability and
> > > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks everyone for the comments and feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It seems to me that the main question here is
> > > about -
> > > > > > "how
> > > > > > > can
> > > > > > > > > > the
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1) I think how and where to host this
> integration
> > > is
> > > > > kind
> > > > > > > of
> > > > > > > > > less
> > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > here. I believe there can be many ways to
> achieve
> > > it.
> > > > > > > > > > > > > > > As part of the contribution, what we are
> looking
> > > for
> > > > > here
> > > > > > > is
> > > > > > > > > how
> > > > > > > > > > > > these
> > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > communities can build the collaboration
> > > relationship
> > > > on
> > > > > > > > > > developing
> > > > > > > > > > > > > > > the integration between Pulsar and Flink. Even
> we
> > > can
> > > > > try
> > > > > > > our
> > > > > > > > > > best
> > > > > > > > > > > to
> > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > up all the updates in Flink community. We are
> still
> > > > > > > > > > > > > > > facing the fact that we have less experiences
> in
> > > > Flink
> > > > > > than
> > > > > > > > > folks
> > > > > > > > > > > in
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > community. In order to make sure we maintain
> and
> > > > > deliver
> > > > > > > > > > > > > > > a high-quality pulsar-flink integration to the
> > > users
> > > > > who
> > > > > > > use
> > > > > > > > > both
> > > > > > > > > > > > > > > technologies, we need some help from the
> experts
> > > from
> > > > > > Flink
> > > > > > > > > > > > community.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a while.
> > > > > Originally
> > > > > > > we
> > > > > > > > > were
> > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > of contributing the connectors back after
> > > integrating
> > > > > > with
> > > > > > > the
> > > > > > > > > > > > > > > new API introduced in FLIP-27. But we decided
> to
> > > > > initiate
> > > > > > > the
> > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > as early as possible. Because we believe there
> are
> > > > more
> > > > > > > > > benefits
> > > > > > > > > > > > doing
> > > > > > > > > > > > > > > it now rather than later. As part of
> contribution,
> > > it
> > > > > can
> > > > > > > help
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > community understand more about Pulsar and the
> > > > > potential
> > > > > > > > > > > integration
> > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > Also we can also help Flink community verify
> the
> > > new
> > > > > > > connector
> > > > > > > > > > API
> > > > > > > > > > > as
> > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket Qin <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the interest in contributing the
> > > Pulsar
> > > > > > > connector.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In general, I think having Pulsar connector
> with
> > > > > strong
> > > > > > > > > support
> > > > > > > > > > > is
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > valuable addition to Flink. So I am happy the
> > > > > shepherd
> > > > > > > this
> > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > Meanwhile, I would also like to provide some
> > > > context
> > > > > > and
> > > > > > > > > recent
> > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The current way Flink maintains its
> connector has
> > > > hit
> > > > > > the
> > > > > > > > > > > > scalability
> > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > With more and more connectors coming into
> Flink
> > > > repo,
> > > > > > we
> > > > > > > are
> > > > > > > > > > > > facing a
> > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > problems such as long build and testing
> time. To
> > > > > > address
> > > > > > > this
> > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > 1. Split out the connectors into a separate
> > > > > repository.
> > > > > > > This
> > > > > > > > > is
> > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > on hold due to potential solution to shorten
> the
> > > > > build
> > > > > > > time.
> > > > > > > > > > > > > > > > 2. Encourage the connectors to stay as
> ecosystem
> > > > > > project
> > > > > > > > > while
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > to provide good support for functionality and
> > > > > > > compatibility
> > > > > > > > > > > tests.
> > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > has driven to create a Flink Ecosystem
> project
> > > > > website
> > > > > > > and it
> > > > > > > > > > is
> > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Given the above efforts, it would be great to
> > > first
> > > > > see
> > > > > > > if we
> > > > > > > > > > can
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > Pulsar connector as an ecosystem project with
> > > great
> > > > > > > support.
> > > > > > > > > It
> > > > > > > > > > > > would
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > good to hear how the Flink Pulsar connector
> is
> > > > tested
> > > > > > > > > currently
> > > > > > > > > > > to
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > we can learn something to maintain it as an
> > > > ecosystem
> > > > > > > project
> > > > > > > > > > > with
> > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > quality and test coverage. If the quality as
> an
> > > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > is
> > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > to guarantee, we may as well adopt it into
> the
> > > main
> > > > > > repo.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27 where
> we
> > > are
> > > > > > > making
> > > > > > > > > > > changes
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Flink source connector architecture and
> > > interface.
> > > > > This
> > > > > > > > > change
> > > > > > > > > > > will
> > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if we
> are
> > > > going
> > > > > to
> > > > > > > have
> > > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > connector in main repo, I am wondering if we
> > > should
> > > > > > hold
> > > > > > > a
> > > > > > > > > > little
> > > > > > > > > > > > bit
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > let the Pulsar connector adapt to the new
> > > interface
> > > > > to
> > > > > > > avoid
> > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay
> Schepler <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm quite worried that we may end up
> repeating
> > > > > > history.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > There were already 2 attempts at
> contributing a
> > > > > > pulsar
> > > > > > > > > > > connector,
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > of which failed because no committer was
> > > getting
> > > > > > > involved,
> > > > > > > > > > > > despite
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > contributor opening a dedicated discussion
> > > thread
> > > > > > > about the
> > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > beforehand and getting several +1's from
> > > > > committers.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We should really make sure that if we
> > > > > welcome/approve
> > > > > > > such
> > > > > > > > > a
> > > > > > > > > > > > > > > > > contribution it will actually get the
> attention
> > > > it
> > > > > > > > > deserves.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> maintaining
> > > > the
> > > > > > > > > connector
> > > > > > > > > > > > > outside
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > documentation
> > > > > to
> > > > > > > give
> > > > > > > > > it
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > With the upcoming page for sharing
> artifacts
> > > > among
> > > > > > the
> > > > > > > > > > > community
> > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > the state of that anyway?), this may be a
> > > better
> > > > > > > option.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann wrote:
> > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > thanks a lot for starting this discussion
> > > > Yijie.
> > > > > I
> > > > > > > think
> > > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > connector would be a very valuable
> addition
> > > > since
> > > > > > > Pulsar
> > > > > > > > > > > > becomes
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > more popular and it would further expand
> > > > Flink's
> > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > from a project perspective it makes
> sense for
> > > > me
> > > > > to
> > > > > > > place
> > > > > > > > > > the
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > My main concern/question is how can the
> Flink
> > > > > > > community
> > > > > > > > > > > > maintain
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector? We have seen in the past that
> > > > > connectors
> > > > > > > are
> > > > > > > > > > some
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > actively developed components because
> they
> > > need
> > > > > to
> > > > > > be
> > > > > > > > > kept
> > > > > > > > > > in
> > > > > > > > > > > > > sync
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > external system and with Flink. Given
> that
> > > the
> > > > > > Pulsar
> > > > > > > > > > > community
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > to help with maintaining, improving and
> > > > evolving
> > > > > > the
> > > > > > > > > > > connector,
> > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > optimistic that we can achieve this.
> Hence,
> > > +1
> > > > > for
> > > > > > > > > > > contributing
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM Sijie Guo
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > > FLINK-9641
> > > > > and
> > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > let
> > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was created
> for
> > > > > bringing
> > > > > > > > > Pulsar
> > > > > > > > > > as
> > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >> sink for Flink. The integration was done
> > > with
> > > > > > Flink
> > > > > > > > > 1.6.0.
> > > > > > > > > > > We
> > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > >> requests about a year ago and we ended
> up
> > > > > > > maintaining
> > > > > > > > > > those
> > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use Flink to
> > > > process
> > > > > > > event
> > > > > > > > > > > streams
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > >
> > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > ).
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple and
> there
> > > is
> > > > no
> > > > > > > schema
> > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> In the past year, we have made a lot of
> > > > changes
> > > > > in
> > > > > > > > > Pulsar
> > > > > > > > > > > and
> > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> citizen in
> > > > > > Pulsar.
> > > > > > > We
> > > > > > > > > > also
> > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > >> other computing engines for processing
> > > Pulsar
> > > > > > event
> > > > > > > > > > streams
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> It led us to rethink how to integrate
> with
> > > > Flink
> > > > > > in
> > > > > > > the
> > > > > > > > > > best
> > > > > > > > > > > > > way.
> > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink connectors
> from
> > > > the
> > > > > > > ground
> > > > > > > > > up
> > > > > > > > > > > > with
> > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >> bring table API and catalog API as the
> > > > > first-class
> > > > > > > > > citizen
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >> integration. With that being said, in
> the
> > > new
> > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > >> you can register pulsar as a flink
> catalog
> > > and
> > > > > > > query /
> > > > > > > > > > > process
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> This is an example about how to use
> Pulsar
> > > as
> > > > a
> > > > > > > Flink
> > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > > explaining
> > > > > why
> > > > > > we
> > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what are
> the
> > > > > changes
> > > > > > we
> > > > > > > > > made
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a simple
> data
> > > > sink
> > > > > > or
> > > > > > > > > source
> > > > > > > > > > > for
> > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > >> actually can be a fully integrated
> streaming
> > > > > data
> > > > > > > > > storage
> > > > > > > > > > > for
> > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > >> areas (sink, source, schema/catalog and
> > > > state).
> > > > > > The
> > > > > > > > > > > > combination
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > >> and Pulsar can create a great streaming
> > > > > warehouse
> > > > > > > > > > > architecture
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > >> streaming-first, unified data
> processing.
> > > > Since
> > > > > we
> > > > > > > are
> > > > > > > > > > > talking
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > >> contribute Pulsar integration to Flink
> here,
> > > > we
> > > > > > are
> > > > > > > also
> > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> integration
> > > > > with
> > > > > > > Flink
> > > > > > > > > to
> > > > > > > > > > > > help
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Hope this give you a bit more background
> > > about
> > > > > the
> > > > > > > > > pulsar
> > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > >> integration. Let me know what are your
> > > > thoughts.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM Yun
> Tang <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes more and
> more
> > > > > > popular
> > > > > > > > > > > recently
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > >>> to see more people willing to
> contribute to
> > > > > Flink
> > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Before any further discussion, would
> you
> > > > please
> > > > > > > give
> > > > > > > > > some
> > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > >>> the relationship between this thread to
> > > > current
> > > > > > > > > existing
> > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > >>> source [1] and sink [2] connector?
> Will the
> > > > > > > > > contribution
> > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > implementation?
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > [hidden email]>
> > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019 13:57
> > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > [hidden email]
> > > > > >
> > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute Pulsar
> Flink
> > > > > > > connector
> > > > > > > > > back
> > > > > > > > > > > to
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> I would like to open the discussion of
> > > > > > contributing
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache
> Pulsar
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a multi-tenant,
> > > > > > > high-performance
> > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar
> includes
> > > > > > multiple
> > > > > > > > > > features
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > >>> native support for multiple clusters
> in a
> > > > > Pulsar
> > > > > > > > > > instance,
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > >>> seamless geo-replication of messages
> across
> > > > > > > clusters,
> > > > > > > > > > very
> > > > > > > > > > > > low
> > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > > scalability
> > > > to
> > > > > > > over a
> > > > > > > > > > > > million
> > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > >>> and guaranteed message delivery with
> > > > persistent
> > > > > > > message
> > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> Nowadays,
> > > > Pulsar
> > > > > > has
> > > > > > > > > been
> > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink connector
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are
> planning
> > > to
> > > > > > > > > contribute
> > > > > > > > > > is
> > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The main
> > > > features
> > > > > > > are:
> > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source with
> > > > > exactly-once
> > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > >>> - Sink streaming results to Pulsar with
> > > > > > > at-least-once
> > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > >>> would update this to exactly-once as
> well
> > > > when
> > > > > > > Pulsar
> > > > > > > > > > gets
> > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > >>> transaction features ready in its 2.5.0
> > > > > version)
> > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API Type
> > > system
> > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > and
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > >>> automatically (de)serialize messages
> with
> > > the
> > > > > > help
> > > > > > > of
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > >>> - Integrate with Flink new Catalog API
> > > > > > > (FLIP-30[4]),
> > > > > > > > > > which
> > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in
> Table API
> > > > as
> > > > > > > well as
> > > > > > > > > > SQL
> > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > >>> [0]
> > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > >>> [2]
> > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>

[Becket Qin]

Hi Yijie,

Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
thread in the mailing list?

https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process

I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
number collision. Can you change the FLIP number to 72?

Thanks,

Jiangjie (Becket) Qin

On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]> wrote:

> Hi Yijie,
>
> Thanks for sharing the pulsar FLIP.
> Would you mind enabling comments/suggestions on the google doc link? This
> way the contributors from the community can comment on the doc.
>
> Best,
> Rong
>
> On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <[hidden email]>
> wrote:
>
> > Hello everyone,
> >
> > I've drafted a FLIP that describes the current design of the Pulsar
> > connector:
> >
> >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> >
> > Please take a look and let me know what you think.
> >
> > Thanks,
> > Yijie
> >
> > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]> wrote:
> > >
> > > Hi All,
> > >
> > > Sorry for joining the discussion late and thanks Yijie & Sijie for
> > driving
> > > the discussion.
> > > I also think the Pulsar connector would be a very valuable addition to
> > > Flink. I can also help out a bit on the review side :-)
> > >
> > > Regarding the timeline, I also share concerns with Becket on the
> > > relationship between the new Pulsar connector and FLIP-27.
> > > There's also another discussion just started by Stephan on dropping
> Kafka
> > > 9/10 support on next Flink release [1].  Although the situation is
> > somewhat
> > > different, and Kafka 9/10 connector has been in Flink for almost 3-4
> > years,
> > > based on the discussion I am not sure if a major version release is a
> > > requirement for removing old connector supports.
> > >
> > > I think there shouldn't be a blocker if we agree the old connector will
> > be
> > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> stated,
> > it
> > > is easier to contribute the source sooner and adjust it later.
> > > We should also ensure we clearly communicate the message: for example,
> > > putting an experimental flag on the pre-FLIP27 connector page of the
> > > website, documentations, etc. Any other thoughts?
> > >
> > > --
> > > Rong
> > >
> > > [1]
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > >
> > >
> > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]>
> wrote:
> > >
> > > > Technically speaking, removing the old connector code is a backwards
> > > > incompatible change which requires a major version bump, i.e. Flink
> > 2.x.
> > > > Given that we don't have a clear plan on when to have the next major
> > > > version release, it seems unclear how long the old connector code
> will
> > be
> > > > there if we check it in right now. Or will we remove the old
> connector
> > > > without a major version bump? In any case, it sounds not quite user
> > > > friendly to the those who might use the old Pulsar connector. I am
> not
> > sure
> > > > if it is worth these potential problems in order to have the Pulsar
> > source
> > > > connector checked in one or two months earlier.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]>
> wrote:
> > > >
> > > > > Agreed, if we check in the old code, we should make it clear that
> it
> > will
> > > > > be removed as soon as the FLIP-27 based version of the connector is
> > > > there.
> > > > > We should not commit to maintaining the old version, that would be
> > indeed
> > > > > too much overhead.
> > > > >
> > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <[hidden email]>
> > wrote:
> > > > >
> > > > > > Hi Stephan,
> > > > > >
> > > > > > Thanks for the volunteering to help.
> > > > > >
> > > > > > Yes, the overhead would just be review capacity. In fact, I am
> not
> > > > > worrying
> > > > > > too much about the review capacity. That is just a one time cost.
> > My
> > > > > > concern is mainly about the long term burden. Assume we have new
> > source
> > > > > > interface ready in 1.10 with newly added Pulsar connectors in old
> > > > > > interface. Later on if we migrate Pulsar to new source interface,
> > the
> > > > old
> > > > > > Pulsar connector might be deprecated almost immediately after
> > checked
> > > > in,
> > > > > > but we may still have to maintain two code bases. For the
> existing
> > > > > > connectors, we have to do that anyways. But it would be good to
> > avoid
> > > > > > introducing a new connector with the same problem.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <[hidden email]>
> > wrote:
> > > > > >
> > > > > > > Hi all!
> > > > > > >
> > > > > > > Nice to see this lively discussion about the Pulsar connector.
> > > > > > > Some thoughts on the open questions:
> > > > > > >
> > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > >
> > > > > > > Looks like the discussion is more going towards contribution. I
> > think
> > > > > > that
> > > > > > > is good, especially if we think that we want to build a
> similarly
> > > > deep
> > > > > > > integration with Pulsar as we have for example with Kafka. The
> > > > > connector
> > > > > > > already looks like a more thorough connector than many others
> we
> > have
> > > > > in
> > > > > > > the repository.
> > > > > > >
> > > > > > > With either a repo split, or the new build system, I hope that
> > the
> > > > > build
> > > > > > > overhead is not a problem.
> > > > > > >
> > > > > > > ## Committer Support
> > > > > > >
> > > > > > > Becket offered some help already, I can also help a bit. I hope
> > that
> > > > > > > between us, we can cover this.
> > > > > > >
> > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > >
> > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > progress, but
> > > > > > will
> > > > > > > take 2 more months, I would estimate, before it is fully
> > available.
> > > > > > >
> > > > > > > If we want to be on the safe side with the contribution, we
> > should
> > > > > > > contribute the source sooner and adjust it later. That would
> also
> > > > help
> > > > > us
> > > > > > > in case things get crazy towards the 1.10 feature freeze and it
> > would
> > > > > be
> > > > > > > hard to find time to review the new changes.
> > > > > > > What would be the overhead of contributing now? Given that the
> > code
> > > > is
> > > > > > > already there, it looks like it would be only review capacity,
> > right?
> > > > > > >
> > > > > > > Best,
> > > > > > > Stephan
> > > > > > >
> > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone!
> > > > > > > >
> > > > > > > > Thanks for your attention and the promotion of this work.
> > > > > > > >
> > > > > > > > We will prepare a FLIP as soon as possible for more specific
> > > > > > discussions.
> > > > > > > >
> > > > > > > > For FLIP-27, it seems that we have not reached a consensus.
> > > > > Therefore,
> > > > > > > > I will explain all the functionalities of the existing
> > connector in
> > > > > > > > the FLIP (including Source, Sink, and Catalog) to continue
> our
> > > > > > > > discussions in FLIP.
> > > > > > > >
> > > > > > > > Thanks for your kind help.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yijie
> > > > > > > >
> > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Sijie,
> > > > > > > > >
> > > > > > > > > If we agree that the goal is to have Pulsar connector in
> > 1.10,
> > > > how
> > > > > > > about
> > > > > > > > we
> > > > > > > > > do the following:
> > > > > > > > >
> > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main repo
> > as it
> > > > > is a
> > > > > > > new
> > > > > > > > > public interface to Flink main repo.
> > > > > > > > > 1. Start to review the Pulsar sink right away as there is
> no
> > > > change
> > > > > > to
> > > > > > > > the
> > > > > > > > > sink interface so far.
> > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to be
> > code
> > > > > > freeze
> > > > > > > in
> > > > > > > > > late Nov and let's say we give a month to the development
> and
> > > > > review
> > > > > > of
> > > > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct.
> There
> > are
> > > > > > still
> > > > > > > 7
> > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is not
> > ready
> > > > by
> > > > > > late
> > > > > > > > > Oct, we can review and check in Pulsar connector with the
> > > > existing
> > > > > > > source
> > > > > > > > > interface. This means we will have Pulsar connector in
> Flink
> > > > 1.10,
> > > > > > > either
> > > > > > > > > with or without FLIP-27.
> > > > > > > > >
> > > > > > > > > Because we are going to have Pulsar sink and source checked
> > in
> > > > > > > > separately,
> > > > > > > > > it might make sense to have two FLIPs, one for Pulsar sink
> > and
> > > > > > another
> > > > > > > > for
> > > > > > > > > Pulsar source. And we can start the work on Pulsar sink
> right
> > > > away.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > >
> > > > > > > > > > What's the take from Flink community? Shall we wait for
> > FLIP-27
> > > > > or
> > > > > > > > shall we
> > > > > > > > > > proceed to next steps? And what the next steps are? :-)
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Sijie
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I think having a Pulsar connector in Flink can be a
> good
> > > > mutual
> > > > > > > > benefit
> > > > > > > > > > to
> > > > > > > > > > > both communities.
> > > > > > > > > > >
> > > > > > > > > > > Another perspective is that Pulsar connector is the 1st
> > > > > streaming
> > > > > > > > > > connector
> > > > > > > > > > > that integrates with Flink's metadata management system
> > and
> > > > > > Catalog
> > > > > > > > APIs.
> > > > > > > > > > > It'll be cool to see how the integration turns out and
> > > > whether
> > > > > we
> > > > > > > > need to
> > > > > > > > > > > improve Flink Catalog stack, which are currently in
> > Beta, to
> > > > > > cater
> > > > > > > to
> > > > > > > > > > > streaming source/sink. Thus I'm in favor of merging
> > Pulsar
> > > > > > > connector
> > > > > > > > into
> > > > > > > > > > > Flink 1.10.
> > > > > > > > > > >
> > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe one
> > for
> > > > > basic
> > > > > > > > > > > source/sink functionalities and another for schema and
> > > > catalog
> > > > > > > > > > integration,
> > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > >
> > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I
> don't
> > > > think
> > > > > > > > FLIP-27
> > > > > > > > > > > should be a blocker in cases where it cannot make its
> way
> > > > into
> > > > > > 1.10
> > > > > > > > or
> > > > > > > > > > > doesn't leave reasonable amount of time for committers
> to
> > > > > review
> > > > > > or
> > > > > > > > for
> > > > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > > > >
> > > > > > > > > > > Bowen
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Till,
> > > > > > > > > > > >
> > > > > > > > > > > > You are right. It all depends on when the new source
> > > > > interface
> > > > > > is
> > > > > > > > going
> > > > > > > > > > > to
> > > > > > > > > > > > be ready. Personally I think it would be there in
> > about a
> > > > > month
> > > > > > > or
> > > > > > > > so.
> > > > > > > > > > > But
> > > > > > > > > > > > I could be too optimistic. It would also be good to
> > hear
> > > > what
> > > > > > do
> > > > > > > > > > Aljoscha
> > > > > > > > > > > > and Stephan think as they are also involved in
> FLIP-27.
> > > > > > > > > > > >
> > > > > > > > > > > > In general I think we should have Pulsar connector in
> > Flink
> > > > > > 1.10,
> > > > > > > > > > > > preferably with the new source interface. We can also
> > check
> > > > > it
> > > > > > in
> > > > > > > > right
> > > > > > > > > > > now
> > > > > > > > > > > > with old source interface, but I suspect few users
> > will use
> > > > > it
> > > > > > > > before
> > > > > > > > > > the
> > > > > > > > > > > > next official release. Therefore, it seems reasonable
> > to
> > > > > wait a
> > > > > > > > little
> > > > > > > > > > > bit
> > > > > > > > > > > > to see whether we can jump to the new source
> > interface. As
> > > > > long
> > > > > > > as
> > > > > > > > we
> > > > > > > > > > > make
> > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit doesn't
> > seem
> > > > to
> > > > > > hurt
> > > > > > > > much.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm wondering what the problem would be if we
> > committed
> > > > the
> > > > > > > > Pulsar
> > > > > > > > > > > > > connector before the new source interface is ready.
> > If I
> > > > > > > > understood
> > > > > > > > > > it
> > > > > > > > > > > > > correctly, then we need to support the old source
> > > > interface
> > > > > > > > anyway
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > existing connectors. By checking it in early I
> could
> > see
> > > > > the
> > > > > > > > benefit
> > > > > > > > > > > that
> > > > > > > > > > > > > our users could start using the connector earlier.
> > > > > Moreover,
> > > > > > it
> > > > > > > > would
> > > > > > > > > > > > > prevent that the Pulsar integration is being
> delayed
> > in
> > > > > case
> > > > > > > > that the
> > > > > > > > > > > > > source interface should be delayed. The only
> > downside I
> > > > see
> > > > > > is
> > > > > > > > the
> > > > > > > > > > > extra
> > > > > > > > > > > > > review effort and potential fixes which might be
> > > > irrelevant
> > > > > > for
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > source interface implementation. I guess it mainly
> > > > depends
> > > > > on
> > > > > > > how
> > > > > > > > > > > certain
> > > > > > > > > > > > > we are when the new source interface will be ready.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Till
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> Although
> > the
> > > > > FLIP
> > > > > > > > wiki
> > > > > > > > > > and
> > > > > > > > > > > > > > discussion thread has been quiet for some time, a
> > few
> > > > > > > > committer /
> > > > > > > > > > > > > > contributors in Flink community were actually
> > > > prototyping
> > > > > > the
> > > > > > > > > > entire
> > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > We have made some good progress there but want to
> > > > update
> > > > > > the
> > > > > > > > FLIP
> > > > > > > > > > > wiki
> > > > > > > > > > > > > > after the entire thing is verified to work in
> case
> > > > there
> > > > > > are
> > > > > > > > some
> > > > > > > > > > > last
> > > > > > > > > > > > > > minute surprise in the implementation. I don't
> > have an
> > > > > > exact
> > > > > > > > ETA
> > > > > > > > > > yet,
> > > > > > > > > > > > > but I
> > > > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am happy to review the current Flink Pulsar
> > connector
> > > > > and
> > > > > > > > see if
> > > > > > > > > > it
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the
> case
> > that
> > > > > we
> > > > > > > > checked
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar connector with some review efforts and
> > shortly
> > > > > after
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for all the feedback and suggestions!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As Sijie said, the goal of the connector has
> > always
> > > > > been
> > > > > > to
> > > > > > > > > > provide
> > > > > > > > > > > > > > > users with the latest features of both systems
> as
> > > > soon
> > > > > as
> > > > > > > > > > possible.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > > propose to contribute the connector to Flink
> and
> > hope
> > > > > to
> > > > > > > get
> > > > > > > > more
> > > > > > > > > > > > > > > suggestions and feedback from Flink experts to
> > ensure
> > > > > the
> > > > > > > > high
> > > > > > > > > > > > quality
> > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at the
> > > > beginning
> > > > > of
> > > > > > > > > > reworking
> > > > > > > > > > > > > > > the connector implementation based on Flink
> 1.9;
> > we
> > > > > also
> > > > > > > > wanted
> > > > > > > > > > to
> > > > > > > > > > > > > > > build a connector that supports both batch and
> > stream
> > > > > > > > computing
> > > > > > > > > > > based
> > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > However, it has been inactive for some time, so
> > we
> > > > > > decided
> > > > > > > to
> > > > > > > > > > > provide
> > > > > > > > > > > > > > > a connector with most of the new features, such
> > as
> > > > the
> > > > > > new
> > > > > > > > type
> > > > > > > > > > > > system
> > > > > > > > > > > > > > > and the new catalog API first. We will pay
> > attention
> > > > to
> > > > > > the
> > > > > > > > > > > progress
> > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it with
> > the
> > > > > > > connector
> > > > > > > > as
> > > > > > > > > > > soon
> > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding the test status of the connector, we
> > are
> > > > > > > following
> > > > > > > > the
> > > > > > > > > > > > other
> > > > > > > > > > > > > > > connectors' test in Flink repository and aimed
> to
> > > > > provide
> > > > > > > > > > > throughout
> > > > > > > > > > > > > > > tests as we could. We are also happy to hear
> > > > > suggestions
> > > > > > > and
> > > > > > > > > > > > > > > supervision from the Flink community to improve
> > the
> > > > > > > > stability and
> > > > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks everyone for the comments and
> feedback.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It seems to me that the main question here is
> > > > about -
> > > > > > > "how
> > > > > > > > can
> > > > > > > > > > > the
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 1) I think how and where to host this
> > integration
> > > > is
> > > > > > kind
> > > > > > > > of
> > > > > > > > > > less
> > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > here. I believe there can be many ways to
> > achieve
> > > > it.
> > > > > > > > > > > > > > > > As part of the contribution, what we are
> > looking
> > > > for
> > > > > > here
> > > > > > > > is
> > > > > > > > > > how
> > > > > > > > > > > > > these
> > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > communities can build the collaboration
> > > > relationship
> > > > > on
> > > > > > > > > > > developing
> > > > > > > > > > > > > > > > the integration between Pulsar and Flink.
> Even
> > we
> > > > can
> > > > > > try
> > > > > > > > our
> > > > > > > > > > > best
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > up all the updates in Flink community. We are
> > still
> > > > > > > > > > > > > > > > facing the fact that we have less experiences
> > in
> > > > > Flink
> > > > > > > than
> > > > > > > > > > folks
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community. In order to make sure we maintain
> > and
> > > > > > deliver
> > > > > > > > > > > > > > > > a high-quality pulsar-flink integration to
> the
> > > > users
> > > > > > who
> > > > > > > > use
> > > > > > > > > > both
> > > > > > > > > > > > > > > > technologies, we need some help from the
> > experts
> > > > from
> > > > > > > Flink
> > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> while.
> > > > > > Originally
> > > > > > > > we
> > > > > > > > > > were
> > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > of contributing the connectors back after
> > > > integrating
> > > > > > > with
> > > > > > > > the
> > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we decided
> > to
> > > > > > initiate
> > > > > > > > the
> > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > as early as possible. Because we believe
> there
> > are
> > > > > more
> > > > > > > > > > benefits
> > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > it now rather than later. As part of
> > contribution,
> > > > it
> > > > > > can
> > > > > > > > help
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community understand more about Pulsar and
> the
> > > > > > potential
> > > > > > > > > > > > integration
> > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > Also we can also help Flink community verify
> > the
> > > > new
> > > > > > > > connector
> > > > > > > > > > > API
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket Qin <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the interest in contributing the
> > > > Pulsar
> > > > > > > > connector.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > In general, I think having Pulsar connector
> > with
> > > > > > strong
> > > > > > > > > > support
> > > > > > > > > > > > is
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > valuable addition to Flink. So I am happy
> the
> > > > > > shepherd
> > > > > > > > this
> > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > Meanwhile, I would also like to provide
> some
> > > > > context
> > > > > > > and
> > > > > > > > > > recent
> > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > The current way Flink maintains its
> > connector has
> > > > > hit
> > > > > > > the
> > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > With more and more connectors coming into
> > Flink
> > > > > repo,
> > > > > > > we
> > > > > > > > are
> > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > problems such as long build and testing
> > time. To
> > > > > > > address
> > > > > > > > this
> > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > 1. Split out the connectors into a separate
> > > > > > repository.
> > > > > > > > This
> > > > > > > > > > is
> > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > on hold due to potential solution to
> shorten
> > the
> > > > > > build
> > > > > > > > time.
> > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay as
> > ecosystem
> > > > > > > project
> > > > > > > > > > while
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > to provide good support for functionality
> and
> > > > > > > > compatibility
> > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > has driven to create a Flink Ecosystem
> > project
> > > > > > website
> > > > > > > > and it
> > > > > > > > > > > is
> > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Given the above efforts, it would be great
> to
> > > > first
> > > > > > see
> > > > > > > > if we
> > > > > > > > > > > can
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem project
> with
> > > > great
> > > > > > > > support.
> > > > > > > > > > It
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar connector
> > is
> > > > > tested
> > > > > > > > > > currently
> > > > > > > > > > > > to
> > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > we can learn something to maintain it as an
> > > > > ecosystem
> > > > > > > > project
> > > > > > > > > > > > with
> > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > quality and test coverage. If the quality
> as
> > an
> > > > > > > ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it into
> > the
> > > > main
> > > > > > > repo.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27
> where
> > we
> > > > are
> > > > > > > > making
> > > > > > > > > > > > changes
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Flink source connector architecture and
> > > > interface.
> > > > > > This
> > > > > > > > > > change
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if we
> > are
> > > > > going
> > > > > > to
> > > > > > > > have
> > > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > connector in main repo, I am wondering if
> we
> > > > should
> > > > > > > hold
> > > > > > > > a
> > > > > > > > > > > little
> > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the new
> > > > interface
> > > > > > to
> > > > > > > > avoid
> > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay
> > Schepler <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm quite worried that we may end up
> > repeating
> > > > > > > history.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > contributing a
> > > > > > > pulsar
> > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > of which failed because no committer was
> > > > getting
> > > > > > > > involved,
> > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > contributor opening a dedicated
> discussion
> > > > thread
> > > > > > > > about the
> > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > beforehand and getting several +1's from
> > > > > > committers.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We should really make sure that if we
> > > > > > welcome/approve
> > > > > > > > such
> > > > > > > > > > a
> > > > > > > > > > > > > > > > > > contribution it will actually get the
> > attention
> > > > > it
> > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > maintaining
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > documentation
> > > > > > to
> > > > > > > > give
> > > > > > > > > > it
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > artifacts
> > > > > among
> > > > > > > the
> > > > > > > > > > > > community
> > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > the state of that anyway?), this may be a
> > > > better
> > > > > > > > option.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann wrote:
> > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> discussion
> > > > > Yijie.
> > > > > > I
> > > > > > > > think
> > > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > connector would be a very valuable
> > addition
> > > > > since
> > > > > > > > Pulsar
> > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > more popular and it would further
> expand
> > > > > Flink's
> > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > from a project perspective it makes
> > sense for
> > > > > me
> > > > > > to
> > > > > > > > place
> > > > > > > > > > > the
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > My main concern/question is how can the
> > Flink
> > > > > > > > community
> > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector? We have seen in the past
> that
> > > > > > connectors
> > > > > > > > are
> > > > > > > > > > > some
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > actively developed components because
> > they
> > > > need
> > > > > > to
> > > > > > > be
> > > > > > > > > > kept
> > > > > > > > > > > in
> > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > external system and with Flink. Given
> > that
> > > > the
> > > > > > > Pulsar
> > > > > > > > > > > > community
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > to help with maintaining, improving and
> > > > > evolving
> > > > > > > the
> > > > > > > > > > > > connector,
> > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > optimistic that we can achieve this.
> > Hence,
> > > > +1
> > > > > > for
> > > > > > > > > > > > contributing
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM Sijie
> Guo
> > <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > > > FLINK-9641
> > > > > > and
> > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was created
> > for
> > > > > > bringing
> > > > > > > > > > Pulsar
> > > > > > > > > > > as
> > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > >> sink for Flink. The integration was
> done
> > > > with
> > > > > > > Flink
> > > > > > > > > > 1.6.0.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > >> requests about a year ago and we ended
> > up
> > > > > > > > maintaining
> > > > > > > > > > > those
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use Flink
> to
> > > > > process
> > > > > > > > event
> > > > > > > > > > > > streams
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > >
> > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple and
> > there
> > > > is
> > > > > no
> > > > > > > > schema
> > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> In the past year, we have made a lot
> of
> > > > > changes
> > > > > > in
> > > > > > > > > > Pulsar
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > citizen in
> > > > > > > Pulsar.
> > > > > > > > We
> > > > > > > > > > > also
> > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > >> other computing engines for processing
> > > > Pulsar
> > > > > > > event
> > > > > > > > > > > streams
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> It led us to rethink how to integrate
> > with
> > > > > Flink
> > > > > > > in
> > > > > > > > the
> > > > > > > > > > > best
> > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> connectors
> > from
> > > > > the
> > > > > > > > ground
> > > > > > > > > > up
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > >> bring table API and catalog API as the
> > > > > > first-class
> > > > > > > > > > citizen
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >> integration. With that being said, in
> > the
> > > > new
> > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > >> you can register pulsar as a flink
> > catalog
> > > > and
> > > > > > > > query /
> > > > > > > > > > > > process
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> This is an example about how to use
> > Pulsar
> > > > as
> > > > > a
> > > > > > > > Flink
> > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > > > explaining
> > > > > > why
> > > > > > > we
> > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what are
> > the
> > > > > > changes
> > > > > > > we
> > > > > > > > > > made
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a simple
> > data
> > > > > sink
> > > > > > > or
> > > > > > > > > > source
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > >> actually can be a fully integrated
> > streaming
> > > > > > data
> > > > > > > > > > storage
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > >> areas (sink, source, schema/catalog
> and
> > > > > state).
> > > > > > > The
> > > > > > > > > > > > > combination
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> streaming
> > > > > > warehouse
> > > > > > > > > > > > architecture
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > processing.
> > > > > Since
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > talking
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to Flink
> > here,
> > > > > we
> > > > > > > are
> > > > > > > > also
> > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > integration
> > > > > > with
> > > > > > > > Flink
> > > > > > > > > > to
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> background
> > > > about
> > > > > > the
> > > > > > > > > > pulsar
> > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > >> integration. Let me know what are your
> > > > > thoughts.
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM Yun
> > Tang <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes more
> and
> > more
> > > > > > > popular
> > > > > > > > > > > > recently
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > contribute to
> > > > > > Flink
> > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> Before any further discussion, would
> > you
> > > > > please
> > > > > > > > give
> > > > > > > > > > some
> > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > >>> the relationship between this thread
> to
> > > > > current
> > > > > > > > > > existing
> > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > >>> source [1] and sink [2] connector?
> > Will the
> > > > > > > > > > contribution
> > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > implementation?
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > [hidden email]>
> > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019
> 13:57
> > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute Pulsar
> > Flink
> > > > > > > > connector
> > > > > > > > > > back
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> I would like to open the discussion
> of
> > > > > > > contributing
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache
> > Pulsar
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a multi-tenant,
> > > > > > > > high-performance
> > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar
> > includes
> > > > > > > multiple
> > > > > > > > > > > features
> > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > >>> native support for multiple clusters
> > in a
> > > > > > Pulsar
> > > > > > > > > > > instance,
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > >>> seamless geo-replication of messages
> > across
> > > > > > > > clusters,
> > > > > > > > > > > very
> > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > > > scalability
> > > > > to
> > > > > > > > over a
> > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery with
> > > > > persistent
> > > > > > > > message
> > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > Nowadays,
> > > > > Pulsar
> > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> connector
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are
> > planning
> > > > to
> > > > > > > > > > contribute
> > > > > > > > > > > is
> > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The
> main
> > > > > features
> > > > > > > > are:
> > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source with
> > > > > > exactly-once
> > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > >>> - Sink streaming results to Pulsar
> with
> > > > > > > > at-least-once
> > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > >>> would update this to exactly-once as
> > well
> > > > > when
> > > > > > > > Pulsar
> > > > > > > > > > > gets
> > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > >>> transaction features ready in its
> 2.5.0
> > > > > > version)
> > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API Type
> > > > system
> > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > >>> automatically (de)serialize messages
> > with
> > > > the
> > > > > > > help
> > > > > > > > of
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new Catalog
> API
> > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > which
> > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in
> > Table API
> > > > > as
> > > > > > > > well as
> > > > > > > > > > > SQL
> > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > >>> [2]
> > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
>

[Stephan Ewen]

Some quick thoughts on the connector contribution process. I basically
reiterate here what Thomas mentioned in another thread about the Kinesis
connector.

For connectors, we should favor a low-overhead contribution process, and
accept user code and changes more readily than in the core system.
That is because connectors have both a big variety of scenarios they get
used in (only through use and many small contributions do they become
really useful over time) and at the same time, and committers do not use
the connector themselves and usually cannot foresee too well what is needed.

Further more, a missing connector (or connector feature) is often a bigger
show stopper for users than a missing API or system feature.

Along these lines of thougt, the conclusion would be to take the Pulsar
connector now, focus the review on legal/dependencies/rough code style and
conventions, label it as "beta" (in the sense of "new code" that is "not
yet tested through longer use") and go ahead. And then evolve it quickly
without putting formal blockers in the way, meaning also adding a new FLIP
27 version when it is there.

Best,
Stephan



On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]> wrote:

> Hi Yijie,
>
> Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> thread in the mailing list?
>
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
>
> I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
> number collision. Can you change the FLIP number to 72?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]> wrote:
>
> > Hi Yijie,
> >
> > Thanks for sharing the pulsar FLIP.
> > Would you mind enabling comments/suggestions on the google doc link? This
> > way the contributors from the community can comment on the doc.
> >
> > Best,
> > Rong
> >
> > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <[hidden email]>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I've drafted a FLIP that describes the current design of the Pulsar
> > > connector:
> > >
> > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > >
> > > Please take a look and let me know what you think.
> > >
> > > Thanks,
> > > Yijie
> > >
> > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Sorry for joining the discussion late and thanks Yijie & Sijie for
> > > driving
> > > > the discussion.
> > > > I also think the Pulsar connector would be a very valuable addition
> to
> > > > Flink. I can also help out a bit on the review side :-)
> > > >
> > > > Regarding the timeline, I also share concerns with Becket on the
> > > > relationship between the new Pulsar connector and FLIP-27.
> > > > There's also another discussion just started by Stephan on dropping
> > Kafka
> > > > 9/10 support on next Flink release [1].  Although the situation is
> > > somewhat
> > > > different, and Kafka 9/10 connector has been in Flink for almost 3-4
> > > years,
> > > > based on the discussion I am not sure if a major version release is a
> > > > requirement for removing old connector supports.
> > > >
> > > > I think there shouldn't be a blocker if we agree the old connector
> will
> > > be
> > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > stated,
> > > it
> > > > is easier to contribute the source sooner and adjust it later.
> > > > We should also ensure we clearly communicate the message: for
> example,
> > > > putting an experimental flag on the pre-FLIP27 connector page of the
> > > > website, documentations, etc. Any other thoughts?
> > > >
> > > > --
> > > > Rong
> > > >
> > > > [1]
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > >
> > > >
> > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]>
> > wrote:
> > > >
> > > > > Technically speaking, removing the old connector code is a
> backwards
> > > > > incompatible change which requires a major version bump, i.e. Flink
> > > 2.x.
> > > > > Given that we don't have a clear plan on when to have the next
> major
> > > > > version release, it seems unclear how long the old connector code
> > will
> > > be
> > > > > there if we check it in right now. Or will we remove the old
> > connector
> > > > > without a major version bump? In any case, it sounds not quite user
> > > > > friendly to the those who might use the old Pulsar connector. I am
> > not
> > > sure
> > > > > if it is worth these potential problems in order to have the Pulsar
> > > source
> > > > > connector checked in one or two months earlier.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]>
> > wrote:
> > > > >
> > > > > > Agreed, if we check in the old code, we should make it clear that
> > it
> > > will
> > > > > > be removed as soon as the FLIP-27 based version of the connector
> is
> > > > > there.
> > > > > > We should not commit to maintaining the old version, that would
> be
> > > indeed
> > > > > > too much overhead.
> > > > > >
> > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <[hidden email]
> >
> > > wrote:
> > > > > >
> > > > > > > Hi Stephan,
> > > > > > >
> > > > > > > Thanks for the volunteering to help.
> > > > > > >
> > > > > > > Yes, the overhead would just be review capacity. In fact, I am
> > not
> > > > > > worrying
> > > > > > > too much about the review capacity. That is just a one time
> cost.
> > > My
> > > > > > > concern is mainly about the long term burden. Assume we have
> new
> > > source
> > > > > > > interface ready in 1.10 with newly added Pulsar connectors in
> old
> > > > > > > interface. Later on if we migrate Pulsar to new source
> interface,
> > > the
> > > > > old
> > > > > > > Pulsar connector might be deprecated almost immediately after
> > > checked
> > > > > in,
> > > > > > > but we may still have to maintain two code bases. For the
> > existing
> > > > > > > connectors, we have to do that anyways. But it would be good to
> > > avoid
> > > > > > > introducing a new connector with the same problem.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <[hidden email]
> >
> > > wrote:
> > > > > > >
> > > > > > > > Hi all!
> > > > > > > >
> > > > > > > > Nice to see this lively discussion about the Pulsar
> connector.
> > > > > > > > Some thoughts on the open questions:
> > > > > > > >
> > > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > > >
> > > > > > > > Looks like the discussion is more going towards
> contribution. I
> > > think
> > > > > > > that
> > > > > > > > is good, especially if we think that we want to build a
> > similarly
> > > > > deep
> > > > > > > > integration with Pulsar as we have for example with Kafka.
> The
> > > > > > connector
> > > > > > > > already looks like a more thorough connector than many others
> > we
> > > have
> > > > > > in
> > > > > > > > the repository.
> > > > > > > >
> > > > > > > > With either a repo split, or the new build system, I hope
> that
> > > the
> > > > > > build
> > > > > > > > overhead is not a problem.
> > > > > > > >
> > > > > > > > ## Committer Support
> > > > > > > >
> > > > > > > > Becket offered some help already, I can also help a bit. I
> hope
> > > that
> > > > > > > > between us, we can cover this.
> > > > > > > >
> > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > >
> > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > progress, but
> > > > > > > will
> > > > > > > > take 2 more months, I would estimate, before it is fully
> > > available.
> > > > > > > >
> > > > > > > > If we want to be on the safe side with the contribution, we
> > > should
> > > > > > > > contribute the source sooner and adjust it later. That would
> > also
> > > > > help
> > > > > > us
> > > > > > > > in case things get crazy towards the 1.10 feature freeze and
> it
> > > would
> > > > > > be
> > > > > > > > hard to find time to review the new changes.
> > > > > > > > What would be the overhead of contributing now? Given that
> the
> > > code
> > > > > is
> > > > > > > > already there, it looks like it would be only review
> capacity,
> > > right?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone!
> > > > > > > > >
> > > > > > > > > Thanks for your attention and the promotion of this work.
> > > > > > > > >
> > > > > > > > > We will prepare a FLIP as soon as possible for more
> specific
> > > > > > > discussions.
> > > > > > > > >
> > > > > > > > > For FLIP-27, it seems that we have not reached a consensus.
> > > > > > Therefore,
> > > > > > > > > I will explain all the functionalities of the existing
> > > connector in
> > > > > > > > > the FLIP (including Source, Sink, and Catalog) to continue
> > our
> > > > > > > > > discussions in FLIP.
> > > > > > > > >
> > > > > > > > > Thanks for your kind help.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Yijie
> > > > > > > > >
> > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Sijie,
> > > > > > > > > >
> > > > > > > > > > If we agree that the goal is to have Pulsar connector in
> > > 1.10,
> > > > > how
> > > > > > > > about
> > > > > > > > > we
> > > > > > > > > > do the following:
> > > > > > > > > >
> > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main
> repo
> > > as it
> > > > > > is a
> > > > > > > > new
> > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > 1. Start to review the Pulsar sink right away as there is
> > no
> > > > > change
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > sink interface so far.
> > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to
> be
> > > code
> > > > > > > freeze
> > > > > > > > in
> > > > > > > > > > late Nov and let's say we give a month to the development
> > and
> > > > > > review
> > > > > > > of
> > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct.
> > There
> > > are
> > > > > > > still
> > > > > > > > 7
> > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is not
> > > ready
> > > > > by
> > > > > > > late
> > > > > > > > > > Oct, we can review and check in Pulsar connector with the
> > > > > existing
> > > > > > > > source
> > > > > > > > > > interface. This means we will have Pulsar connector in
> > Flink
> > > > > 1.10,
> > > > > > > > either
> > > > > > > > > > with or without FLIP-27.
> > > > > > > > > >
> > > > > > > > > > Because we are going to have Pulsar sink and source
> checked
> > > in
> > > > > > > > > separately,
> > > > > > > > > > it might make sense to have two FLIPs, one for Pulsar
> sink
> > > and
> > > > > > > another
> > > > > > > > > for
> > > > > > > > > > Pulsar source. And we can start the work on Pulsar sink
> > right
> > > > > away.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > >
> > > > > > > > > > > What's the take from Flink community? Shall we wait for
> > > FLIP-27
> > > > > > or
> > > > > > > > > shall we
> > > > > > > > > > > proceed to next steps? And what the next steps are? :-)
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Sijie
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I think having a Pulsar connector in Flink can be a
> > good
> > > > > mutual
> > > > > > > > > benefit
> > > > > > > > > > > to
> > > > > > > > > > > > both communities.
> > > > > > > > > > > >
> > > > > > > > > > > > Another perspective is that Pulsar connector is the
> 1st
> > > > > > streaming
> > > > > > > > > > > connector
> > > > > > > > > > > > that integrates with Flink's metadata management
> system
> > > and
> > > > > > > Catalog
> > > > > > > > > APIs.
> > > > > > > > > > > > It'll be cool to see how the integration turns out
> and
> > > > > whether
> > > > > > we
> > > > > > > > > need to
> > > > > > > > > > > > improve Flink Catalog stack, which are currently in
> > > Beta, to
> > > > > > > cater
> > > > > > > > to
> > > > > > > > > > > > streaming source/sink. Thus I'm in favor of merging
> > > Pulsar
> > > > > > > > connector
> > > > > > > > > into
> > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > >
> > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe
> one
> > > for
> > > > > > basic
> > > > > > > > > > > > source/sink functionalities and another for schema
> and
> > > > > catalog
> > > > > > > > > > > integration,
> > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > >
> > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I
> > don't
> > > > > think
> > > > > > > > > FLIP-27
> > > > > > > > > > > > should be a blocker in cases where it cannot make its
> > way
> > > > > into
> > > > > > > 1.10
> > > > > > > > > or
> > > > > > > > > > > > doesn't leave reasonable amount of time for
> committers
> > to
> > > > > > review
> > > > > > > or
> > > > > > > > > for
> > > > > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > > > > >
> > > > > > > > > > > > Bowen
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > >
> > > > > > > > > > > > > You are right. It all depends on when the new
> source
> > > > > > interface
> > > > > > > is
> > > > > > > > > going
> > > > > > > > > > > > to
> > > > > > > > > > > > > be ready. Personally I think it would be there in
> > > about a
> > > > > > month
> > > > > > > > or
> > > > > > > > > so.
> > > > > > > > > > > > But
> > > > > > > > > > > > > I could be too optimistic. It would also be good to
> > > hear
> > > > > what
> > > > > > > do
> > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > and Stephan think as they are also involved in
> > FLIP-27.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In general I think we should have Pulsar connector
> in
> > > Flink
> > > > > > > 1.10,
> > > > > > > > > > > > > preferably with the new source interface. We can
> also
> > > check
> > > > > > it
> > > > > > > in
> > > > > > > > > right
> > > > > > > > > > > > now
> > > > > > > > > > > > > with old source interface, but I suspect few users
> > > will use
> > > > > > it
> > > > > > > > > before
> > > > > > > > > > > the
> > > > > > > > > > > > > next official release. Therefore, it seems
> reasonable
> > > to
> > > > > > wait a
> > > > > > > > > little
> > > > > > > > > > > > bit
> > > > > > > > > > > > > to see whether we can jump to the new source
> > > interface. As
> > > > > > long
> > > > > > > > as
> > > > > > > > > we
> > > > > > > > > > > > make
> > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> doesn't
> > > seem
> > > > > to
> > > > > > > hurt
> > > > > > > > > much.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm wondering what the problem would be if we
> > > committed
> > > > > the
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > connector before the new source interface is
> ready.
> > > If I
> > > > > > > > > understood
> > > > > > > > > > > it
> > > > > > > > > > > > > > correctly, then we need to support the old source
> > > > > interface
> > > > > > > > > anyway
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > existing connectors. By checking it in early I
> > could
> > > see
> > > > > > the
> > > > > > > > > benefit
> > > > > > > > > > > > that
> > > > > > > > > > > > > > our users could start using the connector
> earlier.
> > > > > > Moreover,
> > > > > > > it
> > > > > > > > > would
> > > > > > > > > > > > > > prevent that the Pulsar integration is being
> > delayed
> > > in
> > > > > > case
> > > > > > > > > that the
> > > > > > > > > > > > > > source interface should be delayed. The only
> > > downside I
> > > > > see
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > > > extra
> > > > > > > > > > > > > > review effort and potential fixes which might be
> > > > > irrelevant
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > > source interface implementation. I guess it
> mainly
> > > > > depends
> > > > > > on
> > > > > > > > how
> > > > > > > > > > > > certain
> > > > > > > > > > > > > > we are when the new source interface will be
> ready.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Till
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > Although
> > > the
> > > > > > FLIP
> > > > > > > > > wiki
> > > > > > > > > > > and
> > > > > > > > > > > > > > > discussion thread has been quiet for some
> time, a
> > > few
> > > > > > > > > committer /
> > > > > > > > > > > > > > > contributors in Flink community were actually
> > > > > prototyping
> > > > > > > the
> > > > > > > > > > > entire
> > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > We have made some good progress there but want
> to
> > > > > update
> > > > > > > the
> > > > > > > > > FLIP
> > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > after the entire thing is verified to work in
> > case
> > > > > there
> > > > > > > are
> > > > > > > > > some
> > > > > > > > > > > > last
> > > > > > > > > > > > > > > minute surprise in the implementation. I don't
> > > have an
> > > > > > > exact
> > > > > > > > > ETA
> > > > > > > > > > > yet,
> > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I am happy to review the current Flink Pulsar
> > > connector
> > > > > > and
> > > > > > > > > see if
> > > > > > > > > > > it
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the
> > case
> > > that
> > > > > > we
> > > > > > > > > checked
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > Pulsar connector with some review efforts and
> > > shortly
> > > > > > after
> > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for all the feedback and suggestions!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As Sijie said, the goal of the connector has
> > > always
> > > > > > been
> > > > > > > to
> > > > > > > > > > > provide
> > > > > > > > > > > > > > > > users with the latest features of both
> systems
> > as
> > > > > soon
> > > > > > as
> > > > > > > > > > > possible.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > propose to contribute the connector to Flink
> > and
> > > hope
> > > > > > to
> > > > > > > > get
> > > > > > > > > more
> > > > > > > > > > > > > > > > suggestions and feedback from Flink experts
> to
> > > ensure
> > > > > > the
> > > > > > > > > high
> > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at the
> > > > > beginning
> > > > > > of
> > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > the connector implementation based on Flink
> > 1.9;
> > > we
> > > > > > also
> > > > > > > > > wanted
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > build a connector that supports both batch
> and
> > > stream
> > > > > > > > > computing
> > > > > > > > > > > > based
> > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > However, it has been inactive for some time,
> so
> > > we
> > > > > > > decided
> > > > > > > > to
> > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > a connector with most of the new features,
> such
> > > as
> > > > > the
> > > > > > > new
> > > > > > > > > type
> > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > and the new catalog API first. We will pay
> > > attention
> > > > > to
> > > > > > > the
> > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it
> with
> > > the
> > > > > > > > connector
> > > > > > > > > as
> > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regarding the test status of the connector,
> we
> > > are
> > > > > > > > following
> > > > > > > > > the
> > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > connectors' test in Flink repository and
> aimed
> > to
> > > > > > provide
> > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > tests as we could. We are also happy to hear
> > > > > > suggestions
> > > > > > > > and
> > > > > > > > > > > > > > > > supervision from the Flink community to
> improve
> > > the
> > > > > > > > > stability and
> > > > > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > feedback.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It seems to me that the main question here
> is
> > > > > about -
> > > > > > > > "how
> > > > > > > > > can
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > integration
> > > > > is
> > > > > > > kind
> > > > > > > > > of
> > > > > > > > > > > less
> > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > here. I believe there can be many ways to
> > > achieve
> > > > > it.
> > > > > > > > > > > > > > > > > As part of the contribution, what we are
> > > looking
> > > > > for
> > > > > > > here
> > > > > > > > > is
> > > > > > > > > > > how
> > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > communities can build the collaboration
> > > > > relationship
> > > > > > on
> > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > the integration between Pulsar and Flink.
> > Even
> > > we
> > > > > can
> > > > > > > try
> > > > > > > > > our
> > > > > > > > > > > > best
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > up all the updates in Flink community. We
> are
> > > still
> > > > > > > > > > > > > > > > > facing the fact that we have less
> experiences
> > > in
> > > > > > Flink
> > > > > > > > than
> > > > > > > > > > > folks
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > community. In order to make sure we
> maintain
> > > and
> > > > > > > deliver
> > > > > > > > > > > > > > > > > a high-quality pulsar-flink integration to
> > the
> > > > > users
> > > > > > > who
> > > > > > > > > use
> > > > > > > > > > > both
> > > > > > > > > > > > > > > > > technologies, we need some help from the
> > > experts
> > > > > from
> > > > > > > > Flink
> > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> > while.
> > > > > > > Originally
> > > > > > > > > we
> > > > > > > > > > > were
> > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > of contributing the connectors back after
> > > > > integrating
> > > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> decided
> > > to
> > > > > > > initiate
> > > > > > > > > the
> > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > as early as possible. Because we believe
> > there
> > > are
> > > > > > more
> > > > > > > > > > > benefits
> > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > contribution,
> > > > > it
> > > > > > > can
> > > > > > > > > help
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > community understand more about Pulsar and
> > the
> > > > > > > potential
> > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > Also we can also help Flink community
> verify
> > > the
> > > > > new
> > > > > > > > > connector
> > > > > > > > > > > > API
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket Qin <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for the interest in contributing
> the
> > > > > Pulsar
> > > > > > > > > connector.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> connector
> > > with
> > > > > > > strong
> > > > > > > > > > > support
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am happy
> > the
> > > > > > > shepherd
> > > > > > > > > this
> > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > Meanwhile, I would also like to provide
> > some
> > > > > > context
> > > > > > > > and
> > > > > > > > > > > recent
> > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > connector has
> > > > > > hit
> > > > > > > > the
> > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > With more and more connectors coming into
> > > Flink
> > > > > > repo,
> > > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > problems such as long build and testing
> > > time. To
> > > > > > > > address
> > > > > > > > > this
> > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> separate
> > > > > > > repository.
> > > > > > > > > This
> > > > > > > > > > > is
> > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > on hold due to potential solution to
> > shorten
> > > the
> > > > > > > build
> > > > > > > > > time.
> > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay as
> > > ecosystem
> > > > > > > > project
> > > > > > > > > > > while
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > to provide good support for functionality
> > and
> > > > > > > > > compatibility
> > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > has driven to create a Flink Ecosystem
> > > project
> > > > > > > website
> > > > > > > > > and it
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Given the above efforts, it would be
> great
> > to
> > > > > first
> > > > > > > see
> > > > > > > > > if we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem project
> > with
> > > > > great
> > > > > > > > > support.
> > > > > > > > > > > It
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> connector
> > > is
> > > > > > tested
> > > > > > > > > > > currently
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > we can learn something to maintain it as
> an
> > > > > > ecosystem
> > > > > > > > > project
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > quality and test coverage. If the quality
> > as
> > > an
> > > > > > > > ecosystem
> > > > > > > > > > > > project
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it
> into
> > > the
> > > > > main
> > > > > > > > repo.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27
> > where
> > > we
> > > > > are
> > > > > > > > > making
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Flink source connector architecture and
> > > > > interface.
> > > > > > > This
> > > > > > > > > > > change
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if
> we
> > > are
> > > > > > going
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > connector in main repo, I am wondering if
> > we
> > > > > should
> > > > > > > > hold
> > > > > > > > > a
> > > > > > > > > > > > little
> > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the new
> > > > > interface
> > > > > > > to
> > > > > > > > > avoid
> > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay
> > > Schepler <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm quite worried that we may end up
> > > repeating
> > > > > > > > history.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > contributing a
> > > > > > > > pulsar
> > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > of which failed because no committer
> was
> > > > > getting
> > > > > > > > > involved,
> > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > discussion
> > > > > thread
> > > > > > > > > about the
> > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > beforehand and getting several +1's
> from
> > > > > > > committers.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > We should really make sure that if we
> > > > > > > welcome/approve
> > > > > > > > > such
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > contribution it will actually get the
> > > attention
> > > > > > it
> > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > > maintaining
> > > > > > the
> > > > > > > > > > > connector
> > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > > documentation
> > > > > > > to
> > > > > > > > > give
> > > > > > > > > > > it
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > > artifacts
> > > > > > among
> > > > > > > > the
> > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > the state of that anyway?), this may
> be a
> > > > > better
> > > > > > > > > option.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann
> wrote:
> > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > discussion
> > > > > > Yijie.
> > > > > > > I
> > > > > > > > > think
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > connector would be a very valuable
> > > addition
> > > > > > since
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > more popular and it would further
> > expand
> > > > > > Flink's
> > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > from a project perspective it makes
> > > sense for
> > > > > > me
> > > > > > > to
> > > > > > > > > place
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > My main concern/question is how can
> the
> > > Flink
> > > > > > > > > community
> > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector? We have seen in the past
> > that
> > > > > > > connectors
> > > > > > > > > are
> > > > > > > > > > > > some
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > actively developed components because
> > > they
> > > > > need
> > > > > > > to
> > > > > > > > be
> > > > > > > > > > > kept
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > external system and with Flink. Given
> > > that
> > > > > the
> > > > > > > > Pulsar
> > > > > > > > > > > > > community
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > to help with maintaining, improving
> and
> > > > > > evolving
> > > > > > > > the
> > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > optimistic that we can achieve this.
> > > Hence,
> > > > > +1
> > > > > > > for
> > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM Sijie
> > Guo
> > > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > > > > FLINK-9641
> > > > > > > and
> > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> created
> > > for
> > > > > > > bringing
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >> sink for Flink. The integration was
> > done
> > > > > with
> > > > > > > > Flink
> > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > >> requests about a year ago and we
> ended
> > > up
> > > > > > > > > maintaining
> > > > > > > > > > > > those
> > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use Flink
> > to
> > > > > > process
> > > > > > > > > event
> > > > > > > > > > > > > streams
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > >
> > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple and
> > > there
> > > > > is
> > > > > > no
> > > > > > > > > schema
> > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> In the past year, we have made a lot
> > of
> > > > > > changes
> > > > > > > in
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > > citizen in
> > > > > > > > Pulsar.
> > > > > > > > > We
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > >> other computing engines for
> processing
> > > > > Pulsar
> > > > > > > > event
> > > > > > > > > > > > streams
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> integrate
> > > with
> > > > > > Flink
> > > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > best
> > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > connectors
> > > from
> > > > > > the
> > > > > > > > > ground
> > > > > > > > > > > up
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > >> bring table API and catalog API as
> the
> > > > > > > first-class
> > > > > > > > > > > citizen
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >> integration. With that being said,
> in
> > > the
> > > > > new
> > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > >> you can register pulsar as a flink
> > > catalog
> > > > > and
> > > > > > > > > query /
> > > > > > > > > > > > > process
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> This is an example about how to use
> > > Pulsar
> > > > > as
> > > > > > a
> > > > > > > > > Flink
> > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > > > > explaining
> > > > > > > why
> > > > > > > > we
> > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what
> are
> > > the
> > > > > > > changes
> > > > > > > > we
> > > > > > > > > > > made
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a
> simple
> > > data
> > > > > > sink
> > > > > > > > or
> > > > > > > > > > > source
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > >> actually can be a fully integrated
> > > streaming
> > > > > > > data
> > > > > > > > > > > storage
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > >> areas (sink, source, schema/catalog
> > and
> > > > > > state).
> > > > > > > > The
> > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > streaming
> > > > > > > warehouse
> > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > processing.
> > > > > > Since
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to
> Flink
> > > here,
> > > > > > we
> > > > > > > > are
> > > > > > > > > also
> > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > > integration
> > > > > > > with
> > > > > > > > > Flink
> > > > > > > > > > > to
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > background
> > > > > about
> > > > > > > the
> > > > > > > > > > > pulsar
> > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > >> integration. Let me know what are
> your
> > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM Yun
> > > Tang <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes more
> > and
> > > more
> > > > > > > > popular
> > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > contribute to
> > > > > > > Flink
> > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> Before any further discussion,
> would
> > > you
> > > > > > please
> > > > > > > > > give
> > > > > > > > > > > some
> > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > >>> the relationship between this
> thread
> > to
> > > > > > current
> > > > > > > > > > > existing
> > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2] connector?
> > > Will the
> > > > > > > > > > > contribution
> > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > > implementation?
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019
> > 13:57
> > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> Pulsar
> > > Flink
> > > > > > > > > connector
> > > > > > > > > > > back
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> I would like to open the discussion
> > of
> > > > > > > > contributing
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache
> > > Pulsar
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a multi-tenant,
> > > > > > > > > high-performance
> > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar
> > > includes
> > > > > > > > multiple
> > > > > > > > > > > > features
> > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > >>> native support for multiple
> clusters
> > > in a
> > > > > > > Pulsar
> > > > > > > > > > > > instance,
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> messages
> > > across
> > > > > > > > > clusters,
> > > > > > > > > > > > very
> > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > > > > scalability
> > > > > > to
> > > > > > > > > over a
> > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery
> with
> > > > > > persistent
> > > > > > > > > message
> > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > > Nowadays,
> > > > > > Pulsar
> > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > connector
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are
> > > planning
> > > > > to
> > > > > > > > > > > contribute
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The
> > main
> > > > > > features
> > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source with
> > > > > > > exactly-once
> > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to Pulsar
> > with
> > > > > > > > > at-least-once
> > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > >>> would update this to exactly-once
> as
> > > well
> > > > > > when
> > > > > > > > > Pulsar
> > > > > > > > > > > > gets
> > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > >>> transaction features ready in its
> > 2.5.0
> > > > > > > version)
> > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API
> Type
> > > > > system
> > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> messages
> > > with
> > > > > the
> > > > > > > > help
> > > > > > > > > of
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new Catalog
> > API
> > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > which
> > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in
> > > Table API
> > > > > > as
> > > > > > > > > well as
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>

[Fabian Hueske-2]

+1 to what Stephan (and Thomas) said.

Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen <[hidden email]>:

> Some quick thoughts on the connector contribution process. I basically
> reiterate here what Thomas mentioned in another thread about the Kinesis
> connector.
>
> For connectors, we should favor a low-overhead contribution process, and
> accept user code and changes more readily than in the core system.
> That is because connectors have both a big variety of scenarios they get
> used in (only through use and many small contributions do they become
> really useful over time) and at the same time, and committers do not use
> the connector themselves and usually cannot foresee too well what is
> needed.
>
> Further more, a missing connector (or connector feature) is often a bigger
> show stopper for users than a missing API or system feature.
>
> Along these lines of thougt, the conclusion would be to take the Pulsar
> connector now, focus the review on legal/dependencies/rough code style and
> conventions, label it as "beta" (in the sense of "new code" that is "not
> yet tested through longer use") and go ahead. And then evolve it quickly
> without putting formal blockers in the way, meaning also adding a new FLIP
> 27 version when it is there.
>
> Best,
> Stephan
>
>
>
> On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]> wrote:
>
> > Hi Yijie,
> >
> > Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> > thread in the mailing list?
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> >
> > I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
> > number collision. Can you change the FLIP number to 72?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]> wrote:
> >
> > > Hi Yijie,
> > >
> > > Thanks for sharing the pulsar FLIP.
> > > Would you mind enabling comments/suggestions on the google doc link?
> This
> > > way the contributors from the community can comment on the doc.
> > >
> > > Best,
> > > Rong
> > >
> > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <[hidden email]>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I've drafted a FLIP that describes the current design of the Pulsar
> > > > connector:
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > >
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thanks,
> > > > Yijie
> > > >
> > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Sorry for joining the discussion late and thanks Yijie & Sijie for
> > > > driving
> > > > > the discussion.
> > > > > I also think the Pulsar connector would be a very valuable addition
> > to
> > > > > Flink. I can also help out a bit on the review side :-)
> > > > >
> > > > > Regarding the timeline, I also share concerns with Becket on the
> > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > There's also another discussion just started by Stephan on dropping
> > > Kafka
> > > > > 9/10 support on next Flink release [1].  Although the situation is
> > > > somewhat
> > > > > different, and Kafka 9/10 connector has been in Flink for almost
> 3-4
> > > > years,
> > > > > based on the discussion I am not sure if a major version release
> is a
> > > > > requirement for removing old connector supports.
> > > > >
> > > > > I think there shouldn't be a blocker if we agree the old connector
> > will
> > > > be
> > > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > > stated,
> > > > it
> > > > > is easier to contribute the source sooner and adjust it later.
> > > > > We should also ensure we clearly communicate the message: for
> > example,
> > > > > putting an experimental flag on the pre-FLIP27 connector page of
> the
> > > > > website, documentations, etc. Any other thoughts?
> > > > >
> > > > > --
> > > > > Rong
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > >
> > > > >
> > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Technically speaking, removing the old connector code is a
> > backwards
> > > > > > incompatible change which requires a major version bump, i.e.
> Flink
> > > > 2.x.
> > > > > > Given that we don't have a clear plan on when to have the next
> > major
> > > > > > version release, it seems unclear how long the old connector code
> > > will
> > > > be
> > > > > > there if we check it in right now. Or will we remove the old
> > > connector
> > > > > > without a major version bump? In any case, it sounds not quite
> user
> > > > > > friendly to the those who might use the old Pulsar connector. I
> am
> > > not
> > > > sure
> > > > > > if it is worth these potential problems in order to have the
> Pulsar
> > > > source
> > > > > > connector checked in one or two months earlier.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > > Agreed, if we check in the old code, we should make it clear
> that
> > > it
> > > > will
> > > > > > > be removed as soon as the FLIP-27 based version of the
> connector
> > is
> > > > > > there.
> > > > > > > We should not commit to maintaining the old version, that would
> > be
> > > > indeed
> > > > > > > too much overhead.
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> [hidden email]
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Stephan,
> > > > > > > >
> > > > > > > > Thanks for the volunteering to help.
> > > > > > > >
> > > > > > > > Yes, the overhead would just be review capacity. In fact, I
> am
> > > not
> > > > > > > worrying
> > > > > > > > too much about the review capacity. That is just a one time
> > cost.
> > > > My
> > > > > > > > concern is mainly about the long term burden. Assume we have
> > new
> > > > source
> > > > > > > > interface ready in 1.10 with newly added Pulsar connectors in
> > old
> > > > > > > > interface. Later on if we migrate Pulsar to new source
> > interface,
> > > > the
> > > > > > old
> > > > > > > > Pulsar connector might be deprecated almost immediately after
> > > > checked
> > > > > > in,
> > > > > > > > but we may still have to maintain two code bases. For the
> > > existing
> > > > > > > > connectors, we have to do that anyways. But it would be good
> to
> > > > avoid
> > > > > > > > introducing a new connector with the same problem.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> [hidden email]
> > >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all!
> > > > > > > > >
> > > > > > > > > Nice to see this lively discussion about the Pulsar
> > connector.
> > > > > > > > > Some thoughts on the open questions:
> > > > > > > > >
> > > > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > > > >
> > > > > > > > > Looks like the discussion is more going towards
> > contribution. I
> > > > think
> > > > > > > > that
> > > > > > > > > is good, especially if we think that we want to build a
> > > similarly
> > > > > > deep
> > > > > > > > > integration with Pulsar as we have for example with Kafka.
> > The
> > > > > > > connector
> > > > > > > > > already looks like a more thorough connector than many
> others
> > > we
> > > > have
> > > > > > > in
> > > > > > > > > the repository.
> > > > > > > > >
> > > > > > > > > With either a repo split, or the new build system, I hope
> > that
> > > > the
> > > > > > > build
> > > > > > > > > overhead is not a problem.
> > > > > > > > >
> > > > > > > > > ## Committer Support
> > > > > > > > >
> > > > > > > > > Becket offered some help already, I can also help a bit. I
> > hope
> > > > that
> > > > > > > > > between us, we can cover this.
> > > > > > > > >
> > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > >
> > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > progress, but
> > > > > > > > will
> > > > > > > > > take 2 more months, I would estimate, before it is fully
> > > > available.
> > > > > > > > >
> > > > > > > > > If we want to be on the safe side with the contribution, we
> > > > should
> > > > > > > > > contribute the source sooner and adjust it later. That
> would
> > > also
> > > > > > help
> > > > > > > us
> > > > > > > > > in case things get crazy towards the 1.10 feature freeze
> and
> > it
> > > > would
> > > > > > > be
> > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > What would be the overhead of contributing now? Given that
> > the
> > > > code
> > > > > > is
> > > > > > > > > already there, it looks like it would be only review
> > capacity,
> > > > right?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone!
> > > > > > > > > >
> > > > > > > > > > Thanks for your attention and the promotion of this work.
> > > > > > > > > >
> > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > specific
> > > > > > > > discussions.
> > > > > > > > > >
> > > > > > > > > > For FLIP-27, it seems that we have not reached a
> consensus.
> > > > > > > Therefore,
> > > > > > > > > > I will explain all the functionalities of the existing
> > > > connector in
> > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> continue
> > > our
> > > > > > > > > > discussions in FLIP.
> > > > > > > > > >
> > > > > > > > > > Thanks for your kind help.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yijie
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Sijie,
> > > > > > > > > > >
> > > > > > > > > > > If we agree that the goal is to have Pulsar connector
> in
> > > > 1.10,
> > > > > > how
> > > > > > > > > about
> > > > > > > > > > we
> > > > > > > > > > > do the following:
> > > > > > > > > > >
> > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main
> > repo
> > > > as it
> > > > > > > is a
> > > > > > > > > new
> > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > 1. Start to review the Pulsar sink right away as there
> is
> > > no
> > > > > > change
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > sink interface so far.
> > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to
> > be
> > > > code
> > > > > > > > freeze
> > > > > > > > > in
> > > > > > > > > > > late Nov and let's say we give a month to the
> development
> > > and
> > > > > > > review
> > > > > > > > of
> > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct.
> > > There
> > > > are
> > > > > > > > still
> > > > > > > > > 7
> > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is
> not
> > > > ready
> > > > > > by
> > > > > > > > late
> > > > > > > > > > > Oct, we can review and check in Pulsar connector with
> the
> > > > > > existing
> > > > > > > > > source
> > > > > > > > > > > interface. This means we will have Pulsar connector in
> > > Flink
> > > > > > 1.10,
> > > > > > > > > either
> > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > >
> > > > > > > > > > > Because we are going to have Pulsar sink and source
> > checked
> > > > in
> > > > > > > > > > separately,
> > > > > > > > > > > it might make sense to have two FLIPs, one for Pulsar
> > sink
> > > > and
> > > > > > > > another
> > > > > > > > > > for
> > > > > > > > > > > Pulsar source. And we can start the work on Pulsar sink
> > > right
> > > > > > away.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > >
> > > > > > > > > > > > What's the take from Flink community? Shall we wait
> for
> > > > FLIP-27
> > > > > > > or
> > > > > > > > > > shall we
> > > > > > > > > > > > proceed to next steps? And what the next steps are?
> :-)
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Sijie
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think having a Pulsar connector in Flink can be a
> > > good
> > > > > > mutual
> > > > > > > > > > benefit
> > > > > > > > > > > > to
> > > > > > > > > > > > > both communities.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Another perspective is that Pulsar connector is the
> > 1st
> > > > > > > streaming
> > > > > > > > > > > > connector
> > > > > > > > > > > > > that integrates with Flink's metadata management
> > system
> > > > and
> > > > > > > > Catalog
> > > > > > > > > > APIs.
> > > > > > > > > > > > > It'll be cool to see how the integration turns out
> > and
> > > > > > whether
> > > > > > > we
> > > > > > > > > > need to
> > > > > > > > > > > > > improve Flink Catalog stack, which are currently in
> > > > Beta, to
> > > > > > > > cater
> > > > > > > > > to
> > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of merging
> > > > Pulsar
> > > > > > > > > connector
> > > > > > > > > > into
> > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe
> > one
> > > > for
> > > > > > > basic
> > > > > > > > > > > > > source/sink functionalities and another for schema
> > and
> > > > > > catalog
> > > > > > > > > > > > integration,
> > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I
> > > don't
> > > > > > think
> > > > > > > > > > FLIP-27
> > > > > > > > > > > > > should be a blocker in cases where it cannot make
> its
> > > way
> > > > > > into
> > > > > > > > 1.10
> > > > > > > > > > or
> > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > committers
> > > to
> > > > > > > review
> > > > > > > > or
> > > > > > > > > > for
> > > > > > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Bowen
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You are right. It all depends on when the new
> > source
> > > > > > > interface
> > > > > > > > is
> > > > > > > > > > going
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > be ready. Personally I think it would be there in
> > > > about a
> > > > > > > month
> > > > > > > > > or
> > > > > > > > > > so.
> > > > > > > > > > > > > But
> > > > > > > > > > > > > > I could be too optimistic. It would also be good
> to
> > > > hear
> > > > > > what
> > > > > > > > do
> > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > and Stephan think as they are also involved in
> > > FLIP-27.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In general I think we should have Pulsar
> connector
> > in
> > > > Flink
> > > > > > > > 1.10,
> > > > > > > > > > > > > > preferably with the new source interface. We can
> > also
> > > > check
> > > > > > > it
> > > > > > > > in
> > > > > > > > > > right
> > > > > > > > > > > > > now
> > > > > > > > > > > > > > with old source interface, but I suspect few
> users
> > > > will use
> > > > > > > it
> > > > > > > > > > before
> > > > > > > > > > > > the
> > > > > > > > > > > > > > next official release. Therefore, it seems
> > reasonable
> > > > to
> > > > > > > wait a
> > > > > > > > > > little
> > > > > > > > > > > > > bit
> > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > interface. As
> > > > > > > long
> > > > > > > > > as
> > > > > > > > > > we
> > > > > > > > > > > > > make
> > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > doesn't
> > > > seem
> > > > > > to
> > > > > > > > hurt
> > > > > > > > > > much.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering what the problem would be if we
> > > > committed
> > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > connector before the new source interface is
> > ready.
> > > > If I
> > > > > > > > > > understood
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > correctly, then we need to support the old
> source
> > > > > > interface
> > > > > > > > > > anyway
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > existing connectors. By checking it in early I
> > > could
> > > > see
> > > > > > > the
> > > > > > > > > > benefit
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > our users could start using the connector
> > earlier.
> > > > > > > Moreover,
> > > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > > > > > > prevent that the Pulsar integration is being
> > > delayed
> > > > in
> > > > > > > case
> > > > > > > > > > that the
> > > > > > > > > > > > > > > source interface should be delayed. The only
> > > > downside I
> > > > > > see
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > review effort and potential fixes which might
> be
> > > > > > irrelevant
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > > source interface implementation. I guess it
> > mainly
> > > > > > depends
> > > > > > > on
> > > > > > > > > how
> > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > we are when the new source interface will be
> > ready.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > Although
> > > > the
> > > > > > > FLIP
> > > > > > > > > > wiki
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > time, a
> > > > few
> > > > > > > > > > committer /
> > > > > > > > > > > > > > > > contributors in Flink community were actually
> > > > > > prototyping
> > > > > > > > the
> > > > > > > > > > > > entire
> > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > We have made some good progress there but
> want
> > to
> > > > > > update
> > > > > > > > the
> > > > > > > > > > FLIP
> > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > after the entire thing is verified to work in
> > > case
> > > > > > there
> > > > > > > > are
> > > > > > > > > > some
> > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > minute surprise in the implementation. I
> don't
> > > > have an
> > > > > > > > exact
> > > > > > > > > > ETA
> > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am happy to review the current Flink Pulsar
> > > > connector
> > > > > > > and
> > > > > > > > > > see if
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the
> > > case
> > > > that
> > > > > > > we
> > > > > > > > > > checked
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar connector with some review efforts and
> > > > shortly
> > > > > > > after
> > > > > > > > > > that
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for all the feedback and
> suggestions!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As Sijie said, the goal of the connector
> has
> > > > always
> > > > > > > been
> > > > > > > > to
> > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > users with the latest features of both
> > systems
> > > as
> > > > > > soon
> > > > > > > as
> > > > > > > > > > > > possible.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > propose to contribute the connector to
> Flink
> > > and
> > > > hope
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > more
> > > > > > > > > > > > > > > > > suggestions and feedback from Flink experts
> > to
> > > > ensure
> > > > > > > the
> > > > > > > > > > high
> > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at
> the
> > > > > > beginning
> > > > > > > of
> > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > the connector implementation based on Flink
> > > 1.9;
> > > > we
> > > > > > > also
> > > > > > > > > > wanted
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > build a connector that supports both batch
> > and
> > > > stream
> > > > > > > > > > computing
> > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > However, it has been inactive for some
> time,
> > so
> > > > we
> > > > > > > > decided
> > > > > > > > > to
> > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > a connector with most of the new features,
> > such
> > > > as
> > > > > > the
> > > > > > > > new
> > > > > > > > > > type
> > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > and the new catalog API first. We will pay
> > > > attention
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it
> > with
> > > > the
> > > > > > > > > connector
> > > > > > > > > > as
> > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regarding the test status of the connector,
> > we
> > > > are
> > > > > > > > > following
> > > > > > > > > > the
> > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > connectors' test in Flink repository and
> > aimed
> > > to
> > > > > > > provide
> > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> hear
> > > > > > > suggestions
> > > > > > > > > and
> > > > > > > > > > > > > > > > > supervision from the Flink community to
> > improve
> > > > the
> > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It seems to me that the main question
> here
> > is
> > > > > > about -
> > > > > > > > > "how
> > > > > > > > > > can
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > integration
> > > > > > is
> > > > > > > > kind
> > > > > > > > > > of
> > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > here. I believe there can be many ways to
> > > > achieve
> > > > > > it.
> > > > > > > > > > > > > > > > > > As part of the contribution, what we are
> > > > looking
> > > > > > for
> > > > > > > > here
> > > > > > > > > > is
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > communities can build the collaboration
> > > > > > relationship
> > > > > > > on
> > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > the integration between Pulsar and Flink.
> > > Even
> > > > we
> > > > > > can
> > > > > > > > try
> > > > > > > > > > our
> > > > > > > > > > > > > best
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > up all the updates in Flink community. We
> > are
> > > > still
> > > > > > > > > > > > > > > > > > facing the fact that we have less
> > experiences
> > > > in
> > > > > > > Flink
> > > > > > > > > than
> > > > > > > > > > > > folks
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community. In order to make sure we
> > maintain
> > > > and
> > > > > > > > deliver
> > > > > > > > > > > > > > > > > > a high-quality pulsar-flink integration
> to
> > > the
> > > > > > users
> > > > > > > > who
> > > > > > > > > > use
> > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > technologies, we need some help from the
> > > > experts
> > > > > > from
> > > > > > > > > Flink
> > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> > > while.
> > > > > > > > Originally
> > > > > > > > > > we
> > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > of contributing the connectors back after
> > > > > > integrating
> > > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > decided
> > > > to
> > > > > > > > initiate
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > as early as possible. Because we believe
> > > there
> > > > are
> > > > > > > more
> > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > contribution,
> > > > > > it
> > > > > > > > can
> > > > > > > > > > help
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community understand more about Pulsar
> and
> > > the
> > > > > > > > potential
> > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > verify
> > > > the
> > > > > > new
> > > > > > > > > > connector
> > > > > > > > > > > > > API
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> Qin <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the interest in contributing
> > the
> > > > > > Pulsar
> > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > connector
> > > > with
> > > > > > > > strong
> > > > > > > > > > > > support
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> happy
> > > the
> > > > > > > > shepherd
> > > > > > > > > > this
> > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to provide
> > > some
> > > > > > > context
> > > > > > > > > and
> > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > connector has
> > > > > > > hit
> > > > > > > > > the
> > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > With more and more connectors coming
> into
> > > > Flink
> > > > > > > repo,
> > > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > problems such as long build and testing
> > > > time. To
> > > > > > > > > address
> > > > > > > > > > this
> > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > separate
> > > > > > > > repository.
> > > > > > > > > > This
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > on hold due to potential solution to
> > > shorten
> > > > the
> > > > > > > > build
> > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay as
> > > > ecosystem
> > > > > > > > > project
> > > > > > > > > > > > while
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > to provide good support for
> functionality
> > > and
> > > > > > > > > > compatibility
> > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > has driven to create a Flink Ecosystem
> > > > project
> > > > > > > > website
> > > > > > > > > > and it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Given the above efforts, it would be
> > great
> > > to
> > > > > > first
> > > > > > > > see
> > > > > > > > > > if we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> project
> > > with
> > > > > > great
> > > > > > > > > > support.
> > > > > > > > > > > > It
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > connector
> > > > is
> > > > > > > tested
> > > > > > > > > > > > currently
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > we can learn something to maintain it
> as
> > an
> > > > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> quality
> > > as
> > > > an
> > > > > > > > > ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it
> > into
> > > > the
> > > > > > main
> > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27
> > > where
> > > > we
> > > > > > are
> > > > > > > > > > making
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Flink source connector architecture and
> > > > > > interface.
> > > > > > > > This
> > > > > > > > > > > > change
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if
> > we
> > > > are
> > > > > > > going
> > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > connector in main repo, I am wondering
> if
> > > we
> > > > > > should
> > > > > > > > > hold
> > > > > > > > > > a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the
> new
> > > > > > interface
> > > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay
> > > > Schepler <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm quite worried that we may end up
> > > > repeating
> > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > contributing a
> > > > > > > > > pulsar
> > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > of which failed because no committer
> > was
> > > > > > getting
> > > > > > > > > > involved,
> > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > discussion
> > > > > > thread
> > > > > > > > > > about the
> > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > beforehand and getting several +1's
> > from
> > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > We should really make sure that if we
> > > > > > > > welcome/approve
> > > > > > > > > > such
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > contribution it will actually get the
> > > > attention
> > > > > > > it
> > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > > > maintaining
> > > > > > > the
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > > > documentation
> > > > > > > > to
> > > > > > > > > > give
> > > > > > > > > > > > it
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > > > artifacts
> > > > > > > among
> > > > > > > > > the
> > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > the state of that anyway?), this may
> > be a
> > > > > > better
> > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann
> > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > discussion
> > > > > > > Yijie.
> > > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > connector would be a very valuable
> > > > addition
> > > > > > > since
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > more popular and it would further
> > > expand
> > > > > > > Flink's
> > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > from a project perspective it makes
> > > > sense for
> > > > > > > me
> > > > > > > > to
> > > > > > > > > > place
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > My main concern/question is how can
> > the
> > > > Flink
> > > > > > > > > > community
> > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector? We have seen in the past
> > > that
> > > > > > > > connectors
> > > > > > > > > > are
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > actively developed components
> because
> > > > they
> > > > > > need
> > > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > > kept
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> Given
> > > > that
> > > > > > the
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > to help with maintaining, improving
> > and
> > > > > > > evolving
> > > > > > > > > the
> > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> this.
> > > > Hence,
> > > > > > +1
> > > > > > > > for
> > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> Sijie
> > > Guo
> > > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > > > > > FLINK-9641
> > > > > > > > and
> > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > created
> > > > for
> > > > > > > > bringing
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > >> sink for Flink. The integration
> was
> > > done
> > > > > > with
> > > > > > > > > Flink
> > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > >> requests about a year ago and we
> > ended
> > > > up
> > > > > > > > > > maintaining
> > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> Flink
> > > to
> > > > > > > process
> > > > > > > > > > event
> > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > >
> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple
> and
> > > > there
> > > > > > is
> > > > > > > no
> > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> In the past year, we have made a
> lot
> > > of
> > > > > > > changes
> > > > > > > > in
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > > > citizen in
> > > > > > > > > Pulsar.
> > > > > > > > > > We
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > processing
> > > > > > Pulsar
> > > > > > > > > event
> > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > integrate
> > > > with
> > > > > > > Flink
> > > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > connectors
> > > > from
> > > > > > > the
> > > > > > > > > > ground
> > > > > > > > > > > > up
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > >> bring table API and catalog API as
> > the
> > > > > > > > first-class
> > > > > > > > > > > > citizen
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > >> integration. With that being said,
> > in
> > > > the
> > > > > > new
> > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a flink
> > > > catalog
> > > > > > and
> > > > > > > > > > query /
> > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> This is an example about how to
> use
> > > > Pulsar
> > > > > > as
> > > > > > > a
> > > > > > > > > > Flink
> > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > > > > > explaining
> > > > > > > > why
> > > > > > > > > we
> > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what
> > are
> > > > the
> > > > > > > > changes
> > > > > > > > > we
> > > > > > > > > > > > made
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a
> > simple
> > > > data
> > > > > > > sink
> > > > > > > > > or
> > > > > > > > > > > > source
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > >> actually can be a fully integrated
> > > > streaming
> > > > > > > > data
> > > > > > > > > > > > storage
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> schema/catalog
> > > and
> > > > > > > state).
> > > > > > > > > The
> > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > streaming
> > > > > > > > warehouse
> > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > processing.
> > > > > > > Since
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to
> > Flink
> > > > here,
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > also
> > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > > > integration
> > > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > background
> > > > > > about
> > > > > > > > the
> > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > >> integration. Let me know what are
> > your
> > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM
> Yun
> > > > Tang <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> more
> > > and
> > > > more
> > > > > > > > > popular
> > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > contribute to
> > > > > > > > Flink
> > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Before any further discussion,
> > would
> > > > you
> > > > > > > please
> > > > > > > > > > give
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > thread
> > > to
> > > > > > > current
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> connector?
> > > > Will the
> > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019
> > > 13:57
> > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > Pulsar
> > > > Flink
> > > > > > > > > > connector
> > > > > > > > > > > > back
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> discussion
> > > of
> > > > > > > > > contributing
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache
> > > > Pulsar
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> multi-tenant,
> > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar
> > > > includes
> > > > > > > > > multiple
> > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > clusters
> > > > in a
> > > > > > > > Pulsar
> > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > messages
> > > > across
> > > > > > > > > > clusters,
> > > > > > > > > > > > > very
> > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > > > > > scalability
> > > > > > > to
> > > > > > > > > > over a
> > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery
> > with
> > > > > > > persistent
> > > > > > > > > > message
> > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > > > Nowadays,
> > > > > > > Pulsar
> > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > connector
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are
> > > > planning
> > > > > > to
> > > > > > > > > > > > contribute
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The
> > > main
> > > > > > > features
> > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source
> with
> > > > > > > > exactly-once
> > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> Pulsar
> > > with
> > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > >>> would update this to exactly-once
> > as
> > > > well
> > > > > > > when
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > >>> transaction features ready in its
> > > 2.5.0
> > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API
> > Type
> > > > > > system
> > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > messages
> > > > with
> > > > > > the
> > > > > > > > > help
> > > > > > > > > > of
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> Catalog
> > > API
> > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in
> > > > Table API
> > > > > > > as
> > > > > > > > > > well as
> > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>

[Yijie Shen]

> > > Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> > > thread in the mailing list?

It seems that I don't have permission to create pages in FLIP.


> > > > Thanks for sharing the pulsar FLIP.
> > > > Would you mind enabling comments/suggestions on the google doc link?

Sorry, my bad, I've enabled comments for the doc.
Does it work for now?

On Thu, Sep 19, 2019 at 4:07 PM Fabian Hueske <[hidden email]> wrote:

>
> +1 to what Stephan (and Thomas) said.
>
> Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen <[hidden email]>:
>
> > Some quick thoughts on the connector contribution process. I basically
> > reiterate here what Thomas mentioned in another thread about the Kinesis
> > connector.
> >
> > For connectors, we should favor a low-overhead contribution process, and
> > accept user code and changes more readily than in the core system.
> > That is because connectors have both a big variety of scenarios they get
> > used in (only through use and many small contributions do they become
> > really useful over time) and at the same time, and committers do not use
> > the connector themselves and usually cannot foresee too well what is
> > needed.
> >
> > Further more, a missing connector (or connector feature) is often a bigger
> > show stopper for users than a missing API or system feature.
> >
> > Along these lines of thougt, the conclusion would be to take the Pulsar
> > connector now, focus the review on legal/dependencies/rough code style and
> > conventions, label it as "beta" (in the sense of "new code" that is "not
> > yet tested through longer use") and go ahead. And then evolve it quickly
> > without putting formal blockers in the way, meaning also adding a new FLIP
> > 27 version when it is there.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]> wrote:
> >
> > > Hi Yijie,
> > >
> > > Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> > > thread in the mailing list?
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > >
> > > I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
> > > number collision. Can you change the FLIP number to 72?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]> wrote:
> > >
> > > > Hi Yijie,
> > > >
> > > > Thanks for sharing the pulsar FLIP.
> > > > Would you mind enabling comments/suggestions on the google doc link?
> > This
> > > > way the contributors from the community can comment on the doc.
> > > >
> > > > Best,
> > > > Rong
> > > >
> > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <[hidden email]>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I've drafted a FLIP that describes the current design of the Pulsar
> > > > > connector:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > >
> > > > > Please take a look and let me know what you think.
> > > > >
> > > > > Thanks,
> > > > > Yijie
> > > > >
> > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Sorry for joining the discussion late and thanks Yijie & Sijie for
> > > > > driving
> > > > > > the discussion.
> > > > > > I also think the Pulsar connector would be a very valuable addition
> > > to
> > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > >
> > > > > > Regarding the timeline, I also share concerns with Becket on the
> > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > There's also another discussion just started by Stephan on dropping
> > > > Kafka
> > > > > > 9/10 support on next Flink release [1].  Although the situation is
> > > > > somewhat
> > > > > > different, and Kafka 9/10 connector has been in Flink for almost
> > 3-4
> > > > > years,
> > > > > > based on the discussion I am not sure if a major version release
> > is a
> > > > > > requirement for removing old connector supports.
> > > > > >
> > > > > > I think there shouldn't be a blocker if we agree the old connector
> > > will
> > > > > be
> > > > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > > > stated,
> > > > > it
> > > > > > is easier to contribute the source sooner and adjust it later.
> > > > > > We should also ensure we clearly communicate the message: for
> > > example,
> > > > > > putting an experimental flag on the pre-FLIP27 connector page of
> > the
> > > > > > website, documentations, etc. Any other thoughts?
> > > > > >
> > > > > > --
> > > > > > Rong
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > > > Technically speaking, removing the old connector code is a
> > > backwards
> > > > > > > incompatible change which requires a major version bump, i.e.
> > Flink
> > > > > 2.x.
> > > > > > > Given that we don't have a clear plan on when to have the next
> > > major
> > > > > > > version release, it seems unclear how long the old connector code
> > > > will
> > > > > be
> > > > > > > there if we check it in right now. Or will we remove the old
> > > > connector
> > > > > > > without a major version bump? In any case, it sounds not quite
> > user
> > > > > > > friendly to the those who might use the old Pulsar connector. I
> > am
> > > > not
> > > > > sure
> > > > > > > if it is worth these potential problems in order to have the
> > Pulsar
> > > > > source
> > > > > > > connector checked in one or two months earlier.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]>
> > > > wrote:
> > > > > > >
> > > > > > > > Agreed, if we check in the old code, we should make it clear
> > that
> > > > it
> > > > > will
> > > > > > > > be removed as soon as the FLIP-27 based version of the
> > connector
> > > is
> > > > > > > there.
> > > > > > > > We should not commit to maintaining the old version, that would
> > > be
> > > > > indeed
> > > > > > > > too much overhead.
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Stephan,
> > > > > > > > >
> > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > >
> > > > > > > > > Yes, the overhead would just be review capacity. In fact, I
> > am
> > > > not
> > > > > > > > worrying
> > > > > > > > > too much about the review capacity. That is just a one time
> > > cost.
> > > > > My
> > > > > > > > > concern is mainly about the long term burden. Assume we have
> > > new
> > > > > source
> > > > > > > > > interface ready in 1.10 with newly added Pulsar connectors in
> > > old
> > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > interface,
> > > > > the
> > > > > > > old
> > > > > > > > > Pulsar connector might be deprecated almost immediately after
> > > > > checked
> > > > > > > in,
> > > > > > > > > but we may still have to maintain two code bases. For the
> > > > existing
> > > > > > > > > connectors, we have to do that anyways. But it would be good
> > to
> > > > > avoid
> > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all!
> > > > > > > > > >
> > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > connector.
> > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > >
> > > > > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > > > > >
> > > > > > > > > > Looks like the discussion is more going towards
> > > contribution. I
> > > > > think
> > > > > > > > > that
> > > > > > > > > > is good, especially if we think that we want to build a
> > > > similarly
> > > > > > > deep
> > > > > > > > > > integration with Pulsar as we have for example with Kafka.
> > > The
> > > > > > > > connector
> > > > > > > > > > already looks like a more thorough connector than many
> > others
> > > > we
> > > > > have
> > > > > > > > in
> > > > > > > > > > the repository.
> > > > > > > > > >
> > > > > > > > > > With either a repo split, or the new build system, I hope
> > > that
> > > > > the
> > > > > > > > build
> > > > > > > > > > overhead is not a problem.
> > > > > > > > > >
> > > > > > > > > > ## Committer Support
> > > > > > > > > >
> > > > > > > > > > Becket offered some help already, I can also help a bit. I
> > > hope
> > > > > that
> > > > > > > > > > between us, we can cover this.
> > > > > > > > > >
> > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > >
> > > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > > progress, but
> > > > > > > > > will
> > > > > > > > > > take 2 more months, I would estimate, before it is fully
> > > > > available.
> > > > > > > > > >
> > > > > > > > > > If we want to be on the safe side with the contribution, we
> > > > > should
> > > > > > > > > > contribute the source sooner and adjust it later. That
> > would
> > > > also
> > > > > > > help
> > > > > > > > us
> > > > > > > > > > in case things get crazy towards the 1.10 feature freeze
> > and
> > > it
> > > > > would
> > > > > > > > be
> > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > What would be the overhead of contributing now? Given that
> > > the
> > > > > code
> > > > > > > is
> > > > > > > > > > already there, it looks like it would be only review
> > > capacity,
> > > > > right?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone!
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your attention and the promotion of this work.
> > > > > > > > > > >
> > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > specific
> > > > > > > > > discussions.
> > > > > > > > > > >
> > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > consensus.
> > > > > > > > Therefore,
> > > > > > > > > > > I will explain all the functionalities of the existing
> > > > > connector in
> > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > continue
> > > > our
> > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yijie
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > >
> > > > > > > > > > > > If we agree that the goal is to have Pulsar connector
> > in
> > > > > 1.10,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > we
> > > > > > > > > > > > do the following:
> > > > > > > > > > > >
> > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main
> > > repo
> > > > > as it
> > > > > > > > is a
> > > > > > > > > > new
> > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > 1. Start to review the Pulsar sink right away as there
> > is
> > > > no
> > > > > > > change
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to
> > > be
> > > > > code
> > > > > > > > > freeze
> > > > > > > > > > in
> > > > > > > > > > > > late Nov and let's say we give a month to the
> > development
> > > > and
> > > > > > > > review
> > > > > > > > > of
> > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct.
> > > > There
> > > > > are
> > > > > > > > > still
> > > > > > > > > > 7
> > > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is
> > not
> > > > > ready
> > > > > > > by
> > > > > > > > > late
> > > > > > > > > > > > Oct, we can review and check in Pulsar connector with
> > the
> > > > > > > existing
> > > > > > > > > > source
> > > > > > > > > > > > interface. This means we will have Pulsar connector in
> > > > Flink
> > > > > > > 1.10,
> > > > > > > > > > either
> > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > >
> > > > > > > > > > > > Because we are going to have Pulsar sink and source
> > > checked
> > > > > in
> > > > > > > > > > > separately,
> > > > > > > > > > > > it might make sense to have two FLIPs, one for Pulsar
> > > sink
> > > > > and
> > > > > > > > > another
> > > > > > > > > > > for
> > > > > > > > > > > > Pulsar source. And we can start the work on Pulsar sink
> > > > right
> > > > > > > away.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What's the take from Flink community? Shall we wait
> > for
> > > > > FLIP-27
> > > > > > > > or
> > > > > > > > > > > shall we
> > > > > > > > > > > > > proceed to next steps? And what the next steps are?
> > :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Sijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think having a Pulsar connector in Flink can be a
> > > > good
> > > > > > > mutual
> > > > > > > > > > > benefit
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Another perspective is that Pulsar connector is the
> > > 1st
> > > > > > > > streaming
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > that integrates with Flink's metadata management
> > > system
> > > > > and
> > > > > > > > > Catalog
> > > > > > > > > > > APIs.
> > > > > > > > > > > > > > It'll be cool to see how the integration turns out
> > > and
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > > > need to
> > > > > > > > > > > > > > improve Flink Catalog stack, which are currently in
> > > > > Beta, to
> > > > > > > > > cater
> > > > > > > > > > to
> > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of merging
> > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > into
> > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe
> > > one
> > > > > for
> > > > > > > > basic
> > > > > > > > > > > > > > source/sink functionalities and another for schema
> > > and
> > > > > > > catalog
> > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I
> > > > don't
> > > > > > > think
> > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > should be a blocker in cases where it cannot make
> > its
> > > > way
> > > > > > > into
> > > > > > > > > 1.10
> > > > > > > > > > > or
> > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > committers
> > > > to
> > > > > > > > review
> > > > > > > > > or
> > > > > > > > > > > for
> > > > > > > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You are right. It all depends on when the new
> > > source
> > > > > > > > interface
> > > > > > > > > is
> > > > > > > > > > > going
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > be ready. Personally I think it would be there in
> > > > > about a
> > > > > > > > month
> > > > > > > > > > or
> > > > > > > > > > > so.
> > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > I could be too optimistic. It would also be good
> > to
> > > > > hear
> > > > > > > what
> > > > > > > > > do
> > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > and Stephan think as they are also involved in
> > > > FLIP-27.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In general I think we should have Pulsar
> > connector
> > > in
> > > > > Flink
> > > > > > > > > 1.10,
> > > > > > > > > > > > > > > preferably with the new source interface. We can
> > > also
> > > > > check
> > > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > right
> > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > with old source interface, but I suspect few
> > users
> > > > > will use
> > > > > > > > it
> > > > > > > > > > > before
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > reasonable
> > > > > to
> > > > > > > > wait a
> > > > > > > > > > > little
> > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > > interface. As
> > > > > > > > long
> > > > > > > > > > as
> > > > > > > > > > > we
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > > doesn't
> > > > > seem
> > > > > > > to
> > > > > > > > > hurt
> > > > > > > > > > > much.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm wondering what the problem would be if we
> > > > > committed
> > > > > > > the
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > connector before the new source interface is
> > > ready.
> > > > > If I
> > > > > > > > > > > understood
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > correctly, then we need to support the old
> > source
> > > > > > > interface
> > > > > > > > > > > anyway
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > existing connectors. By checking it in early I
> > > > could
> > > > > see
> > > > > > > > the
> > > > > > > > > > > benefit
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > our users could start using the connector
> > > earlier.
> > > > > > > > Moreover,
> > > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > > > > > > prevent that the Pulsar integration is being
> > > > delayed
> > > > > in
> > > > > > > > case
> > > > > > > > > > > that the
> > > > > > > > > > > > > > > > source interface should be delayed. The only
> > > > > downside I
> > > > > > > see
> > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > review effort and potential fixes which might
> > be
> > > > > > > irrelevant
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > source interface implementation. I guess it
> > > mainly
> > > > > > > depends
> > > > > > > > on
> > > > > > > > > > how
> > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > we are when the new source interface will be
> > > ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > > Although
> > > > > the
> > > > > > > > FLIP
> > > > > > > > > > > wiki
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > > time, a
> > > > > few
> > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > contributors in Flink community were actually
> > > > > > > prototyping
> > > > > > > > > the
> > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > We have made some good progress there but
> > want
> > > to
> > > > > > > update
> > > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > after the entire thing is verified to work in
> > > > case
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > some
> > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > minute surprise in the implementation. I
> > don't
> > > > > have an
> > > > > > > > > exact
> > > > > > > > > > > ETA
> > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I am happy to review the current Flink Pulsar
> > > > > connector
> > > > > > > > and
> > > > > > > > > > > see if
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the
> > > > case
> > > > > that
> > > > > > > > we
> > > > > > > > > > > checked
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar connector with some review efforts and
> > > > > shortly
> > > > > > > > after
> > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > suggestions!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As Sijie said, the goal of the connector
> > has
> > > > > always
> > > > > > > > been
> > > > > > > > > to
> > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > users with the latest features of both
> > > systems
> > > > as
> > > > > > > soon
> > > > > > > > as
> > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > propose to contribute the connector to
> > Flink
> > > > and
> > > > > hope
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > suggestions and feedback from Flink experts
> > > to
> > > > > ensure
> > > > > > > > the
> > > > > > > > > > > high
> > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at
> > the
> > > > > > > beginning
> > > > > > > > of
> > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > the connector implementation based on Flink
> > > > 1.9;
> > > > > we
> > > > > > > > also
> > > > > > > > > > > wanted
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > build a connector that supports both batch
> > > and
> > > > > stream
> > > > > > > > > > > computing
> > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > However, it has been inactive for some
> > time,
> > > so
> > > > > we
> > > > > > > > > decided
> > > > > > > > > > to
> > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > a connector with most of the new features,
> > > such
> > > > > as
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > type
> > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > and the new catalog API first. We will pay
> > > > > attention
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it
> > > with
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > as
> > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regarding the test status of the connector,
> > > we
> > > > > are
> > > > > > > > > > following
> > > > > > > > > > > the
> > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > connectors' test in Flink repository and
> > > aimed
> > > > to
> > > > > > > > provide
> > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> > hear
> > > > > > > > suggestions
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > > supervision from the Flink community to
> > > improve
> > > > > the
> > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > > feedback.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It seems to me that the main question
> > here
> > > is
> > > > > > > about -
> > > > > > > > > > "how
> > > > > > > > > > > can
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > > integration
> > > > > > > is
> > > > > > > > > kind
> > > > > > > > > > > of
> > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > here. I believe there can be many ways to
> > > > > achieve
> > > > > > > it.
> > > > > > > > > > > > > > > > > > > As part of the contribution, what we are
> > > > > looking
> > > > > > > for
> > > > > > > > > here
> > > > > > > > > > > is
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > communities can build the collaboration
> > > > > > > relationship
> > > > > > > > on
> > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > the integration between Pulsar and Flink.
> > > > Even
> > > > > we
> > > > > > > can
> > > > > > > > > try
> > > > > > > > > > > our
> > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > up all the updates in Flink community. We
> > > are
> > > > > still
> > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > experiences
> > > > > in
> > > > > > > > Flink
> > > > > > > > > > than
> > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > maintain
> > > > > and
> > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink integration
> > to
> > > > the
> > > > > > > users
> > > > > > > > > who
> > > > > > > > > > > use
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > technologies, we need some help from the
> > > > > experts
> > > > > > > from
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> > > > while.
> > > > > > > > > Originally
> > > > > > > > > > > we
> > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > of contributing the connectors back after
> > > > > > > integrating
> > > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > > decided
> > > > > to
> > > > > > > > > initiate
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > as early as possible. Because we believe
> > > > there
> > > > > are
> > > > > > > > more
> > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > > contribution,
> > > > > > > it
> > > > > > > > > can
> > > > > > > > > > > help
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community understand more about Pulsar
> > and
> > > > the
> > > > > > > > > potential
> > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > > verify
> > > > > the
> > > > > > > new
> > > > > > > > > > > connector
> > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> > Qin <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the interest in contributing
> > > the
> > > > > > > Pulsar
> > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > connector
> > > > > with
> > > > > > > > > strong
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> > happy
> > > > the
> > > > > > > > > shepherd
> > > > > > > > > > > this
> > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to provide
> > > > some
> > > > > > > > context
> > > > > > > > > > and
> > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > > connector has
> > > > > > > > hit
> > > > > > > > > > the
> > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > With more and more connectors coming
> > into
> > > > > Flink
> > > > > > > > repo,
> > > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > problems such as long build and testing
> > > > > time. To
> > > > > > > > > > address
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > > separate
> > > > > > > > > repository.
> > > > > > > > > > > This
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > on hold due to potential solution to
> > > > shorten
> > > > > the
> > > > > > > > > build
> > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay as
> > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > to provide good support for
> > functionality
> > > > and
> > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > has driven to create a Flink Ecosystem
> > > > > project
> > > > > > > > > website
> > > > > > > > > > > and it
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Given the above efforts, it would be
> > > great
> > > > to
> > > > > > > first
> > > > > > > > > see
> > > > > > > > > > > if we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > project
> > > > with
> > > > > > > great
> > > > > > > > > > > support.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > connector
> > > > > is
> > > > > > > > tested
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > we can learn something to maintain it
> > as
> > > an
> > > > > > > > ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > quality
> > > > as
> > > > > an
> > > > > > > > > > ecosystem
> > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it
> > > into
> > > > > the
> > > > > > > main
> > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27
> > > > where
> > > > > we
> > > > > > > are
> > > > > > > > > > > making
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Flink source connector architecture and
> > > > > > > interface.
> > > > > > > > > This
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if
> > > we
> > > > > are
> > > > > > > > going
> > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > connector in main repo, I am wondering
> > if
> > > > we
> > > > > > > should
> > > > > > > > > > hold
> > > > > > > > > > > a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the
> > new
> > > > > > > interface
> > > > > > > > > to
> > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay
> > > > > Schepler <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may end up
> > > > > repeating
> > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > > contributing a
> > > > > > > > > > pulsar
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > of which failed because no committer
> > > was
> > > > > > > getting
> > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > discussion
> > > > > > > thread
> > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > beforehand and getting several +1's
> > > from
> > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > We should really make sure that if we
> > > > > > > > > welcome/approve
> > > > > > > > > > > such
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > contribution it will actually get the
> > > > > attention
> > > > > > > > it
> > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > > > > maintaining
> > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > > > > documentation
> > > > > > > > > to
> > > > > > > > > > > give
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > > > > artifacts
> > > > > > > > among
> > > > > > > > > > the
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > the state of that anyway?), this may
> > > be a
> > > > > > > better
> > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann
> > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > > discussion
> > > > > > > > Yijie.
> > > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > connector would be a very valuable
> > > > > addition
> > > > > > > > since
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > more popular and it would further
> > > > expand
> > > > > > > > Flink's
> > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > from a project perspective it makes
> > > > > sense for
> > > > > > > > me
> > > > > > > > > to
> > > > > > > > > > > place
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > My main concern/question is how can
> > > the
> > > > > Flink
> > > > > > > > > > > community
> > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector? We have seen in the past
> > > > that
> > > > > > > > > connectors
> > > > > > > > > > > are
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > actively developed components
> > because
> > > > > they
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > > kept
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> > Given
> > > > > that
> > > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > to help with maintaining, improving
> > > and
> > > > > > > > evolving
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> > this.
> > > > > Hence,
> > > > > > > +1
> > > > > > > > > for
> > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> > Sijie
> > > > Guo
> > > > > <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > > > > > > FLINK-9641
> > > > > > > > > and
> > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > > created
> > > > > for
> > > > > > > > > bringing
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The integration
> > was
> > > > done
> > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > >> requests about a year ago and we
> > > ended
> > > > > up
> > > > > > > > > > > maintaining
> > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> > Flink
> > > > to
> > > > > > > > process
> > > > > > > > > > > event
> > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > >
> > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple
> > and
> > > > > there
> > > > > > > is
> > > > > > > > no
> > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> In the past year, we have made a
> > lot
> > > > of
> > > > > > > > changes
> > > > > > > > > in
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > > > > citizen in
> > > > > > > > > > Pulsar.
> > > > > > > > > > > We
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > processing
> > > > > > > Pulsar
> > > > > > > > > > event
> > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > integrate
> > > > > with
> > > > > > > > Flink
> > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > connectors
> > > > > from
> > > > > > > > the
> > > > > > > > > > > ground
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog API as
> > > the
> > > > > > > > > first-class
> > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >> integration. With that being said,
> > > in
> > > > > the
> > > > > > > new
> > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a flink
> > > > > catalog
> > > > > > > and
> > > > > > > > > > > query /
> > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> This is an example about how to
> > use
> > > > > Pulsar
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > > > > > > explaining
> > > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what
> > > are
> > > > > the
> > > > > > > > > changes
> > > > > > > > > > we
> > > > > > > > > > > > > made
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a
> > > simple
> > > > > data
> > > > > > > > sink
> > > > > > > > > > or
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > >> actually can be a fully integrated
> > > > > streaming
> > > > > > > > > data
> > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > schema/catalog
> > > > and
> > > > > > > > state).
> > > > > > > > > > The
> > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > > streaming
> > > > > > > > > warehouse
> > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > > processing.
> > > > > > > > Since
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to
> > > Flink
> > > > > here,
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > > > > integration
> > > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > > background
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > >> integration. Let me know what are
> > > your
> > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM
> > Yun
> > > > > Tang <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> > more
> > > > and
> > > > > more
> > > > > > > > > > popular
> > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > > contribute to
> > > > > > > > > Flink
> > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Before any further discussion,
> > > would
> > > > > you
> > > > > > > > please
> > > > > > > > > > > give
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > > thread
> > > > to
> > > > > > > > current
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > connector?
> > > > > Will the
> > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019
> > > > 13:57
> > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > > Pulsar
> > > > > Flink
> > > > > > > > > > > connector
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > discussion
> > > > of
> > > > > > > > > > contributing
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache
> > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > multi-tenant,
> > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar
> > > > > includes
> > > > > > > > > > multiple
> > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > clusters
> > > > > in a
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > messages
> > > > > across
> > > > > > > > > > > clusters,
> > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > > > > > > scalability
> > > > > > > > to
> > > > > > > > > > > over a
> > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery
> > > with
> > > > > > > > persistent
> > > > > > > > > > > message
> > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > > > > Nowadays,
> > > > > > > > Pulsar
> > > > > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > > connector
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are
> > > > > planning
> > > > > > > to
> > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The
> > > > main
> > > > > > > > features
> > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source
> > with
> > > > > > > > > exactly-once
> > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > Pulsar
> > > > with
> > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > >>> would update this to exactly-once
> > > as
> > > > > well
> > > > > > > > when
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > >>> transaction features ready in its
> > > > 2.5.0
> > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API
> > > Type
> > > > > > > system
> > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > messages
> > > > > with
> > > > > > > the
> > > > > > > > > > help
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > Catalog
> > > > API
> > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in
> > > > > Table API
> > > > > > > > as
> > > > > > > > > > > well as
> > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >

[Fabian Hueske-2]

Hi Yijie,

I can give you permission if you tell me your Confluence user name.

Thanks, Fabian

Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
[hidden email]>:

> > > > Could you please follow the FLIP process to start a new FLIP
> [DISCUSSION]
> > > > thread in the mailing list?
>
> It seems that I don't have permission to create pages in FLIP.
>
>
> > > > > Thanks for sharing the pulsar FLIP.
> > > > > Would you mind enabling comments/suggestions on the google doc
> link?
>
> Sorry, my bad, I've enabled comments for the doc.
> Does it work for now?
>
> On Thu, Sep 19, 2019 at 4:07 PM Fabian Hueske <[hidden email]> wrote:
> >
> > +1 to what Stephan (and Thomas) said.
> >
> > Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen <
> [hidden email]>:
> >
> > > Some quick thoughts on the connector contribution process. I basically
> > > reiterate here what Thomas mentioned in another thread about the
> Kinesis
> > > connector.
> > >
> > > For connectors, we should favor a low-overhead contribution process,
> and
> > > accept user code and changes more readily than in the core system.
> > > That is because connectors have both a big variety of scenarios they
> get
> > > used in (only through use and many small contributions do they become
> > > really useful over time) and at the same time, and committers do not
> use
> > > the connector themselves and usually cannot foresee too well what is
> > > needed.
> > >
> > > Further more, a missing connector (or connector feature) is often a
> bigger
> > > show stopper for users than a missing API or system feature.
> > >
> > > Along these lines of thougt, the conclusion would be to take the Pulsar
> > > connector now, focus the review on legal/dependencies/rough code style
> and
> > > conventions, label it as "beta" (in the sense of "new code" that is
> "not
> > > yet tested through longer use") and go ahead. And then evolve it
> quickly
> > > without putting formal blockers in the way, meaning also adding a new
> FLIP
> > > 27 version when it is there.
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > >
> > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]>
> wrote:
> > >
> > > > Hi Yijie,
> > > >
> > > > Could you please follow the FLIP process to start a new FLIP
> [DISCUSSION]
> > > > thread in the mailing list?
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > >
> > > > I see two FLIP-69 discussion in the mailing list now. So there is a
> FLIP
> > > > number collision. Can you change the FLIP number to 72?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> wrote:
> > > >
> > > > > Hi Yijie,
> > > > >
> > > > > Thanks for sharing the pulsar FLIP.
> > > > > Would you mind enabling comments/suggestions on the google doc
> link?
> > > This
> > > > > way the contributors from the community can comment on the doc.
> > > > >
> > > > > Best,
> > > > > Rong
> > > > >
> > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I've drafted a FLIP that describes the current design of the
> Pulsar
> > > > > > connector:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > >
> > > > > > Please take a look and let me know what you think.
> > > > > >
> > > > > > Thanks,
> > > > > > Yijie
> > > > > >
> > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Sorry for joining the discussion late and thanks Yijie & Sijie
> for
> > > > > > driving
> > > > > > > the discussion.
> > > > > > > I also think the Pulsar connector would be a very valuable
> addition
> > > > to
> > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > >
> > > > > > > Regarding the timeline, I also share concerns with Becket on
> the
> > > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > > There's also another discussion just started by Stephan on
> dropping
> > > > > Kafka
> > > > > > > 9/10 support on next Flink release [1].  Although the
> situation is
> > > > > > somewhat
> > > > > > > different, and Kafka 9/10 connector has been in Flink for
> almost
> > > 3-4
> > > > > > years,
> > > > > > > based on the discussion I am not sure if a major version
> release
> > > is a
> > > > > > > requirement for removing old connector supports.
> > > > > > >
> > > > > > > I think there shouldn't be a blocker if we agree the old
> connector
> > > > will
> > > > > > be
> > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> Stephan
> > > > > stated,
> > > > > > it
> > > > > > > is easier to contribute the source sooner and adjust it later.
> > > > > > > We should also ensure we clearly communicate the message: for
> > > > example,
> > > > > > > putting an experimental flag on the pre-FLIP27 connector page
> of
> > > the
> > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > >
> > > > > > > --
> > > > > > > Rong
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> [hidden email]>
> > > > > wrote:
> > > > > > >
> > > > > > > > Technically speaking, removing the old connector code is a
> > > > backwards
> > > > > > > > incompatible change which requires a major version bump, i.e.
> > > Flink
> > > > > > 2.x.
> > > > > > > > Given that we don't have a clear plan on when to have the
> next
> > > > major
> > > > > > > > version release, it seems unclear how long the old connector
> code
> > > > > will
> > > > > > be
> > > > > > > > there if we check it in right now. Or will we remove the old
> > > > > connector
> > > > > > > > without a major version bump? In any case, it sounds not
> quite
> > > user
> > > > > > > > friendly to the those who might use the old Pulsar
> connector. I
> > > am
> > > > > not
> > > > > > sure
> > > > > > > > if it is worth these potential problems in order to have the
> > > Pulsar
> > > > > > source
> > > > > > > > connector checked in one or two months earlier.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> [hidden email]>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Agreed, if we check in the old code, we should make it
> clear
> > > that
> > > > > it
> > > > > > will
> > > > > > > > > be removed as soon as the FLIP-27 based version of the
> > > connector
> > > > is
> > > > > > > > there.
> > > > > > > > > We should not commit to maintaining the old version, that
> would
> > > > be
> > > > > > indeed
> > > > > > > > > too much overhead.
> > > > > > > > >
> > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Stephan,
> > > > > > > > > >
> > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > >
> > > > > > > > > > Yes, the overhead would just be review capacity. In
> fact, I
> > > am
> > > > > not
> > > > > > > > > worrying
> > > > > > > > > > too much about the review capacity. That is just a one
> time
> > > > cost.
> > > > > > My
> > > > > > > > > > concern is mainly about the long term burden. Assume we
> have
> > > > new
> > > > > > source
> > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> connectors in
> > > > old
> > > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > > interface,
> > > > > > the
> > > > > > > > old
> > > > > > > > > > Pulsar connector might be deprecated almost immediately
> after
> > > > > > checked
> > > > > > > > in,
> > > > > > > > > > but we may still have to maintain two code bases. For the
> > > > > existing
> > > > > > > > > > connectors, we have to do that anyways. But it would be
> good
> > > to
> > > > > > avoid
> > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all!
> > > > > > > > > > >
> > > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > > connector.
> > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > >
> > > > > > > > > > > ## Contribute to Flink or maintain as a community
> package
> > > > > > > > > > >
> > > > > > > > > > > Looks like the discussion is more going towards
> > > > contribution. I
> > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > is good, especially if we think that we want to build a
> > > > > similarly
> > > > > > > > deep
> > > > > > > > > > > integration with Pulsar as we have for example with
> Kafka.
> > > > The
> > > > > > > > > connector
> > > > > > > > > > > already looks like a more thorough connector than many
> > > others
> > > > > we
> > > > > > have
> > > > > > > > > in
> > > > > > > > > > > the repository.
> > > > > > > > > > >
> > > > > > > > > > > With either a repo split, or the new build system, I
> hope
> > > > that
> > > > > > the
> > > > > > > > > build
> > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > >
> > > > > > > > > > > ## Committer Support
> > > > > > > > > > >
> > > > > > > > > > > Becket offered some help already, I can also help a
> bit. I
> > > > hope
> > > > > > that
> > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > >
> > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > >
> > > > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > > > progress, but
> > > > > > > > > > will
> > > > > > > > > > > take 2 more months, I would estimate, before it is
> fully
> > > > > > available.
> > > > > > > > > > >
> > > > > > > > > > > If we want to be on the safe side with the
> contribution, we
> > > > > > should
> > > > > > > > > > > contribute the source sooner and adjust it later. That
> > > would
> > > > > also
> > > > > > > > help
> > > > > > > > > us
> > > > > > > > > > > in case things get crazy towards the 1.10 feature
> freeze
> > > and
> > > > it
> > > > > > would
> > > > > > > > > be
> > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > What would be the overhead of contributing now? Given
> that
> > > > the
> > > > > > code
> > > > > > > > is
> > > > > > > > > > > already there, it looks like it would be only review
> > > > capacity,
> > > > > > right?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Stephan
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your attention and the promotion of this
> work.
> > > > > > > > > > > >
> > > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > > specific
> > > > > > > > > > discussions.
> > > > > > > > > > > >
> > > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > > consensus.
> > > > > > > > > Therefore,
> > > > > > > > > > > > I will explain all the functionalities of the
> existing
> > > > > > connector in
> > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > > continue
> > > > > our
> > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Yijie
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> connector
> > > in
> > > > > > 1.10,
> > > > > > > > how
> > > > > > > > > > > about
> > > > > > > > > > > > we
> > > > > > > > > > > > > do the following:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink
> main
> > > > repo
> > > > > > as it
> > > > > > > > > is a
> > > > > > > > > > > new
> > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > 1. Start to review the Pulsar sink right away as
> there
> > > is
> > > > > no
> > > > > > > > change
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is
> going to
> > > > be
> > > > > > code
> > > > > > > > > > freeze
> > > > > > > > > > > in
> > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > development
> > > > > and
> > > > > > > > > review
> > > > > > > > > > of
> > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late
> Oct.
> > > > > There
> > > > > > are
> > > > > > > > > > still
> > > > > > > > > > > 7
> > > > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27
> is
> > > not
> > > > > > ready
> > > > > > > > by
> > > > > > > > > > late
> > > > > > > > > > > > > Oct, we can review and check in Pulsar connector
> with
> > > the
> > > > > > > > existing
> > > > > > > > > > > source
> > > > > > > > > > > > > interface. This means we will have Pulsar
> connector in
> > > > > Flink
> > > > > > > > 1.10,
> > > > > > > > > > > either
> > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Because we are going to have Pulsar sink and source
> > > > checked
> > > > > > in
> > > > > > > > > > > > separately,
> > > > > > > > > > > > > it might make sense to have two FLIPs, one for
> Pulsar
> > > > sink
> > > > > > and
> > > > > > > > > > another
> > > > > > > > > > > > for
> > > > > > > > > > > > > Pulsar source. And we can start the work on Pulsar
> sink
> > > > > right
> > > > > > > > away.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What's the take from Flink community? Shall we
> wait
> > > for
> > > > > > FLIP-27
> > > > > > > > > or
> > > > > > > > > > > > shall we
> > > > > > > > > > > > > > proceed to next steps? And what the next steps
> are?
> > > :-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think having a Pulsar connector in Flink can
> be a
> > > > > good
> > > > > > > > mutual
> > > > > > > > > > > > benefit
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Another perspective is that Pulsar connector
> is the
> > > > 1st
> > > > > > > > > streaming
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > that integrates with Flink's metadata
> management
> > > > system
> > > > > > and
> > > > > > > > > > Catalog
> > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > It'll be cool to see how the integration turns
> out
> > > > and
> > > > > > > > whether
> > > > > > > > > we
> > > > > > > > > > > > need to
> > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> currently in
> > > > > > Beta, to
> > > > > > > > > > cater
> > > > > > > > > > > to
> > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> merging
> > > > > > Pulsar
> > > > > > > > > > > connector
> > > > > > > > > > > > into
> > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g.
> maybe
> > > > one
> > > > > > for
> > > > > > > > > basic
> > > > > > > > > > > > > > > source/sink functionalities and another for
> schema
> > > > and
> > > > > > > > catalog
> > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27.
> But I
> > > > > don't
> > > > > > > > think
> > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > should be a blocker in cases where it cannot
> make
> > > its
> > > > > way
> > > > > > > > into
> > > > > > > > > > 1.10
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > > committers
> > > > > to
> > > > > > > > > review
> > > > > > > > > > or
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> interfaces.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You are right. It all depends on when the new
> > > > source
> > > > > > > > > interface
> > > > > > > > > > is
> > > > > > > > > > > > going
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > be ready. Personally I think it would be
> there in
> > > > > > about a
> > > > > > > > > month
> > > > > > > > > > > or
> > > > > > > > > > > > so.
> > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > I could be too optimistic. It would also be
> good
> > > to
> > > > > > hear
> > > > > > > > what
> > > > > > > > > > do
> > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > and Stephan think as they are also involved
> in
> > > > > FLIP-27.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In general I think we should have Pulsar
> > > connector
> > > > in
> > > > > > Flink
> > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > preferably with the new source interface. We
> can
> > > > also
> > > > > > check
> > > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > right
> > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > with old source interface, but I suspect few
> > > users
> > > > > > will use
> > > > > > > > > it
> > > > > > > > > > > > before
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > > reasonable
> > > > > > to
> > > > > > > > > wait a
> > > > > > > > > > > > little
> > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > > > interface. As
> > > > > > > > > long
> > > > > > > > > > > as
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > > > doesn't
> > > > > > seem
> > > > > > > > to
> > > > > > > > > > hurt
> > > > > > > > > > > > much.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann
> <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm wondering what the problem would be if
> we
> > > > > > committed
> > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > connector before the new source interface
> is
> > > > ready.
> > > > > > If I
> > > > > > > > > > > > understood
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > correctly, then we need to support the old
> > > source
> > > > > > > > interface
> > > > > > > > > > > > anyway
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > existing connectors. By checking it in
> early I
> > > > > could
> > > > > > see
> > > > > > > > > the
> > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > our users could start using the connector
> > > > earlier.
> > > > > > > > > Moreover,
> > > > > > > > > > it
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > prevent that the Pulsar integration is
> being
> > > > > delayed
> > > > > > in
> > > > > > > > > case
> > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > source interface should be delayed. The
> only
> > > > > > downside I
> > > > > > > > see
> > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > review effort and potential fixes which
> might
> > > be
> > > > > > > > irrelevant
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > source interface implementation. I guess it
> > > > mainly
> > > > > > > > depends
> > > > > > > > > on
> > > > > > > > > > > how
> > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > we are when the new source interface will
> be
> > > > ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > > > Although
> > > > > > the
> > > > > > > > > FLIP
> > > > > > > > > > > > wiki
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > > > time, a
> > > > > > few
> > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > contributors in Flink community were
> actually
> > > > > > > > prototyping
> > > > > > > > > > the
> > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > We have made some good progress there but
> > > want
> > > > to
> > > > > > > > update
> > > > > > > > > > the
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > after the entire thing is verified to
> work in
> > > > > case
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > minute surprise in the implementation. I
> > > don't
> > > > > > have an
> > > > > > > > > > exact
> > > > > > > > > > > > ETA
> > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > guess it is going to be within a month
> or so.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I am happy to review the current Flink
> Pulsar
> > > > > > connector
> > > > > > > > > and
> > > > > > > > > > > > see if
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to
> avoid the
> > > > > case
> > > > > > that
> > > > > > > > > we
> > > > > > > > > > > > checked
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar connector with some review
> efforts and
> > > > > > shortly
> > > > > > > > > after
> > > > > > > > > > > > that
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie
> Shen <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > suggestions!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> connector
> > > has
> > > > > > always
> > > > > > > > > been
> > > > > > > > > > to
> > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > users with the latest features of both
> > > > systems
> > > > > as
> > > > > > > > soon
> > > > > > > > > as
> > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > propose to contribute the connector to
> > > Flink
> > > > > and
> > > > > > hope
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> experts
> > > > to
> > > > > > ensure
> > > > > > > > > the
> > > > > > > > > > > > high
> > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence
> at
> > > the
> > > > > > > > beginning
> > > > > > > > > of
> > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > the connector implementation based on
> Flink
> > > > > 1.9;
> > > > > > we
> > > > > > > > > also
> > > > > > > > > > > > wanted
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > build a connector that supports both
> batch
> > > > and
> > > > > > stream
> > > > > > > > > > > > computing
> > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > However, it has been inactive for some
> > > time,
> > > > so
> > > > > > we
> > > > > > > > > > decided
> > > > > > > > > > > to
> > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > a connector with most of the new
> features,
> > > > such
> > > > > > as
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > type
> > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > and the new catalog API first. We will
> pay
> > > > > > attention
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate
> it
> > > > with
> > > > > > the
> > > > > > > > > > > connector
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Regarding the test status of the
> connector,
> > > > we
> > > > > > are
> > > > > > > > > > > following
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > connectors' test in Flink repository
> and
> > > > aimed
> > > > > to
> > > > > > > > > provide
> > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> > > hear
> > > > > > > > > suggestions
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > supervision from the Flink community to
> > > > improve
> > > > > > the
> > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > performance of the connector
> continuously.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie
> Guo <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > > > feedback.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It seems to me that the main question
> > > here
> > > > is
> > > > > > > > about -
> > > > > > > > > > > "how
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > > > integration
> > > > > > > > is
> > > > > > > > > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > here. I believe there can be many
> ways to
> > > > > > achieve
> > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > As part of the contribution, what we
> are
> > > > > > looking
> > > > > > > > for
> > > > > > > > > > here
> > > > > > > > > > > > is
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > communities can build the
> collaboration
> > > > > > > > relationship
> > > > > > > > > on
> > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > the integration between Pulsar and
> Flink.
> > > > > Even
> > > > > > we
> > > > > > > > can
> > > > > > > > > > try
> > > > > > > > > > > > our
> > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> community. We
> > > > are
> > > > > > still
> > > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > > experiences
> > > > > > in
> > > > > > > > > Flink
> > > > > > > > > > > than
> > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > > maintain
> > > > > > and
> > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> integration
> > > to
> > > > > the
> > > > > > > > users
> > > > > > > > > > who
> > > > > > > > > > > > use
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > technologies, we need some help from
> the
> > > > > > experts
> > > > > > > > from
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27
> for a
> > > > > while.
> > > > > > > > > > Originally
> > > > > > > > > > > > we
> > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > of contributing the connectors back
> after
> > > > > > > > integrating
> > > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > > > decided
> > > > > > to
> > > > > > > > > > initiate
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > as early as possible. Because we
> believe
> > > > > there
> > > > > > are
> > > > > > > > > more
> > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > > > contribution,
> > > > > > > > it
> > > > > > > > > > can
> > > > > > > > > > > > help
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community understand more about
> Pulsar
> > > and
> > > > > the
> > > > > > > > > > potential
> > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > > > verify
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> > > Qin <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> contributing
> > > > the
> > > > > > > > Pulsar
> > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > > connector
> > > > > > with
> > > > > > > > > > strong
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> > > happy
> > > > > the
> > > > > > > > > > shepherd
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> provide
> > > > > some
> > > > > > > > > context
> > > > > > > > > > > and
> > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > > > connector has
> > > > > > > > > hit
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > With more and more connectors
> coming
> > > into
> > > > > > Flink
> > > > > > > > > repo,
> > > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > problems such as long build and
> testing
> > > > > > time. To
> > > > > > > > > > > address
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > > > separate
> > > > > > > > > > repository.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > on hold due to potential solution
> to
> > > > > shorten
> > > > > > the
> > > > > > > > > > build
> > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to
> stay as
> > > > > > ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > functionality
> > > > > and
> > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> Ecosystem
> > > > > > project
> > > > > > > > > > website
> > > > > > > > > > > > and it
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > through some final approval
> process.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Given the above efforts, it would
> be
> > > > great
> > > > > to
> > > > > > > > first
> > > > > > > > > > see
> > > > > > > > > > > > if we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > > project
> > > > > with
> > > > > > > > great
> > > > > > > > > > > > support.
> > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > > connector
> > > > > > is
> > > > > > > > > tested
> > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > we can learn something to maintain
> it
> > > as
> > > > an
> > > > > > > > > ecosystem
> > > > > > > > > > > > project
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > > quality
> > > > > as
> > > > > > an
> > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt
> it
> > > > into
> > > > > > the
> > > > > > > > main
> > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> FLIP-27
> > > > > where
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > making
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Flink source connector
> architecture and
> > > > > > > > interface.
> > > > > > > > > > This
> > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing
> wise, if
> > > > we
> > > > > > are
> > > > > > > > > going
> > > > > > > > > > to
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> wondering
> > > if
> > > > > we
> > > > > > > > should
> > > > > > > > > > > hold
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to
> the
> > > new
> > > > > > > > interface
> > > > > > > > > > to
> > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> Chesnay
> > > > > > Schepler <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may
> end up
> > > > > > repeating
> > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > > > contributing a
> > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > of which failed because no
> committer
> > > > was
> > > > > > > > getting
> > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > > discussion
> > > > > > > > thread
> > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > beforehand and getting several
> +1's
> > > > from
> > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > We should really make sure that
> if we
> > > > > > > > > > welcome/approve
> > > > > > > > > > > > such
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > contribution it will actually
> get the
> > > > > > attention
> > > > > > > > > it
> > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> recommend
> > > > > > maintaining
> > > > > > > > > the
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from
> the
> > > > > > > > documentation
> > > > > > > > > > to
> > > > > > > > > > > > give
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> sharing
> > > > > > artifacts
> > > > > > > > > among
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > the state of that anyway?), this
> may
> > > > be a
> > > > > > > > better
> > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> Rohrmann
> > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > > > discussion
> > > > > > > > > Yijie.
> > > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> valuable
> > > > > > addition
> > > > > > > > > since
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> further
> > > > > expand
> > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > from a project perspective it
> makes
> > > > > > sense for
> > > > > > > > > me
> > > > > > > > > > to
> > > > > > > > > > > > place
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > My main concern/question is
> how can
> > > > the
> > > > > > Flink
> > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in the
> past
> > > > > that
> > > > > > > > > > connectors
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > actively developed components
> > > because
> > > > > > they
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> > > Given
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> improving
> > > > and
> > > > > > > > > evolving
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> > > this.
> > > > > > Hence,
> > > > > > > > +1
> > > > > > > > > > for
> > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> > > Sijie
> > > > > Guo
> > > > > > <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver
> behind
> > > > > > > > FLINK-9641
> > > > > > > > > > and
> > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > > > created
> > > > > > for
> > > > > > > > > > bringing
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> integration
> > > was
> > > > > done
> > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago and
> we
> > > > ended
> > > > > > up
> > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> > > Flink
> > > > > to
> > > > > > > > > process
> > > > > > > > > > > > event
> > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > >
> > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty
> simple
> > > and
> > > > > > there
> > > > > > > > is
> > > > > > > > > no
> > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have
> made a
> > > lot
> > > > > of
> > > > > > > > > changes
> > > > > > > > > > in
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> first-class
> > > > > > citizen in
> > > > > > > > > > > Pulsar.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > > processing
> > > > > > > > Pulsar
> > > > > > > > > > > event
> > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > > integrate
> > > > > > with
> > > > > > > > > Flink
> > > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > > connectors
> > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > ground
> > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog
> API as
> > > > the
> > > > > > > > > > first-class
> > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > >> integration. With that being
> said,
> > > > in
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a
> flink
> > > > > > catalog
> > > > > > > > and
> > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> This is an example about how
> to
> > > use
> > > > > > Pulsar
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog
> post
> > > > > > > > explaining
> > > > > > > > > > why
> > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and
> what
> > > > are
> > > > > > the
> > > > > > > > > > changes
> > > > > > > > > > > we
> > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just
> a
> > > > simple
> > > > > > data
> > > > > > > > > sink
> > > > > > > > > > > or
> > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> integrated
> > > > > > streaming
> > > > > > > > > > data
> > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > schema/catalog
> > > > > and
> > > > > > > > > state).
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > > > streaming
> > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > > > processing.
> > > > > > > > > Since
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration
> to
> > > > Flink
> > > > > > here,
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve
> the
> > > > > > integration
> > > > > > > > > > with
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > > > background
> > > > > > > > about
> > > > > > > > > > the
> > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know what
> are
> > > > your
> > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54
> AM
> > > Yun
> > > > > > Tang <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> > > more
> > > > > and
> > > > > > more
> > > > > > > > > > > popular
> > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > > > contribute to
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> discussion,
> > > > would
> > > > > > you
> > > > > > > > > please
> > > > > > > > > > > > give
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > > > thread
> > > > > to
> > > > > > > > > current
> > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > connector?
> > > > > > Will the
> > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> different
> > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > >>>
> ________________________________
> > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3,
> 2019
> > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > > > Pulsar
> > > > > > Flink
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > > discussion
> > > > > of
> > > > > > > > > > > contributing
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to
> Apache
> > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > multi-tenant,
> > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> Pulsar
> > > > > > includes
> > > > > > > > > > > multiple
> > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > > clusters
> > > > > > in a
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > > messages
> > > > > > across
> > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> seamless
> > > > > > > > scalability
> > > > > > > > > to
> > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> delivery
> > > > with
> > > > > > > > > persistent
> > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> BookKeeper.
> > > > > > Nowadays,
> > > > > > > > > Pulsar
> > > > > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > > > connector
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector
> we are
> > > > > > planning
> > > > > > > > to
> > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar
> 2.4.0. The
> > > > > main
> > > > > > > > > features
> > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> source
> > > with
> > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > > Pulsar
> > > > > with
> > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> exactly-once
> > > > as
> > > > > > well
> > > > > > > > > when
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > >>> transaction features ready
> in its
> > > > > 2.5.0
> > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table
> API
> > > > Type
> > > > > > > > system
> > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > > messages
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > help
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > > Catalog
> > > > > API
> > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> tables in
> > > > > > Table API
> > > > > > > > > as
> > > > > > > > > > > > well as
> > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > >>> [1]
> https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

[Yijie Shen]

Hi Fabian,

It's   yijieshen

Thanks for your help!

On Thu, Sep 19, 2019 at 4:52 PM Fabian Hueske <[hidden email]> wrote:

>
> Hi Yijie,
>
> I can give you permission if you tell me your Confluence user name.
>
> Thanks, Fabian
>
> Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
> [hidden email]>:
>
> > > > > Could you please follow the FLIP process to start a new FLIP
> > [DISCUSSION]
> > > > > thread in the mailing list?
> >
> > It seems that I don't have permission to create pages in FLIP.
> >
> >
> > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > Would you mind enabling comments/suggestions on the google doc
> > link?
> >
> > Sorry, my bad, I've enabled comments for the doc.
> > Does it work for now?
> >
> > On Thu, Sep 19, 2019 at 4:07 PM Fabian Hueske <[hidden email]> wrote:
> > >
> > > +1 to what Stephan (and Thomas) said.
> > >
> > > Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen <
> > [hidden email]>:
> > >
> > > > Some quick thoughts on the connector contribution process. I basically
> > > > reiterate here what Thomas mentioned in another thread about the
> > Kinesis
> > > > connector.
> > > >
> > > > For connectors, we should favor a low-overhead contribution process,
> > and
> > > > accept user code and changes more readily than in the core system.
> > > > That is because connectors have both a big variety of scenarios they
> > get
> > > > used in (only through use and many small contributions do they become
> > > > really useful over time) and at the same time, and committers do not
> > use
> > > > the connector themselves and usually cannot foresee too well what is
> > > > needed.
> > > >
> > > > Further more, a missing connector (or connector feature) is often a
> > bigger
> > > > show stopper for users than a missing API or system feature.
> > > >
> > > > Along these lines of thougt, the conclusion would be to take the Pulsar
> > > > connector now, focus the review on legal/dependencies/rough code style
> > and
> > > > conventions, label it as "beta" (in the sense of "new code" that is
> > "not
> > > > yet tested through longer use") and go ahead. And then evolve it
> > quickly
> > > > without putting formal blockers in the way, meaning also adding a new
> > FLIP
> > > > 27 version when it is there.
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > >
> > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]>
> > wrote:
> > > >
> > > > > Hi Yijie,
> > > > >
> > > > > Could you please follow the FLIP process to start a new FLIP
> > [DISCUSSION]
> > > > > thread in the mailing list?
> > > > >
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > >
> > > > > I see two FLIP-69 discussion in the mailing list now. So there is a
> > FLIP
> > > > > number collision. Can you change the FLIP number to 72?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> > wrote:
> > > > >
> > > > > > Hi Yijie,
> > > > > >
> > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > Would you mind enabling comments/suggestions on the google doc
> > link?
> > > > This
> > > > > > way the contributors from the community can comment on the doc.
> > > > > >
> > > > > > Best,
> > > > > > Rong
> > > > > >
> > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > I've drafted a FLIP that describes the current design of the
> > Pulsar
> > > > > > > connector:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > >
> > > > > > > Please take a look and let me know what you think.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yijie
> > > > > > >
> > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Sorry for joining the discussion late and thanks Yijie & Sijie
> > for
> > > > > > > driving
> > > > > > > > the discussion.
> > > > > > > > I also think the Pulsar connector would be a very valuable
> > addition
> > > > > to
> > > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > > >
> > > > > > > > Regarding the timeline, I also share concerns with Becket on
> > the
> > > > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > > > There's also another discussion just started by Stephan on
> > dropping
> > > > > > Kafka
> > > > > > > > 9/10 support on next Flink release [1].  Although the
> > situation is
> > > > > > > somewhat
> > > > > > > > different, and Kafka 9/10 connector has been in Flink for
> > almost
> > > > 3-4
> > > > > > > years,
> > > > > > > > based on the discussion I am not sure if a major version
> > release
> > > > is a
> > > > > > > > requirement for removing old connector supports.
> > > > > > > >
> > > > > > > > I think there shouldn't be a blocker if we agree the old
> > connector
> > > > > will
> > > > > > > be
> > > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> > Stephan
> > > > > > stated,
> > > > > > > it
> > > > > > > > is easier to contribute the source sooner and adjust it later.
> > > > > > > > We should also ensure we clearly communicate the message: for
> > > > > example,
> > > > > > > > putting an experimental flag on the pre-FLIP27 connector page
> > of
> > > > the
> > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rong
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Technically speaking, removing the old connector code is a
> > > > > backwards
> > > > > > > > > incompatible change which requires a major version bump, i.e.
> > > > Flink
> > > > > > > 2.x.
> > > > > > > > > Given that we don't have a clear plan on when to have the
> > next
> > > > > major
> > > > > > > > > version release, it seems unclear how long the old connector
> > code
> > > > > > will
> > > > > > > be
> > > > > > > > > there if we check it in right now. Or will we remove the old
> > > > > > connector
> > > > > > > > > without a major version bump? In any case, it sounds not
> > quite
> > > > user
> > > > > > > > > friendly to the those who might use the old Pulsar
> > connector. I
> > > > am
> > > > > > not
> > > > > > > sure
> > > > > > > > > if it is worth these potential problems in order to have the
> > > > Pulsar
> > > > > > > source
> > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Agreed, if we check in the old code, we should make it
> > clear
> > > > that
> > > > > > it
> > > > > > > will
> > > > > > > > > > be removed as soon as the FLIP-27 based version of the
> > > > connector
> > > > > is
> > > > > > > > > there.
> > > > > > > > > > We should not commit to maintaining the old version, that
> > would
> > > > > be
> > > > > > > indeed
> > > > > > > > > > too much overhead.
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Stephan,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > >
> > > > > > > > > > > Yes, the overhead would just be review capacity. In
> > fact, I
> > > > am
> > > > > > not
> > > > > > > > > > worrying
> > > > > > > > > > > too much about the review capacity. That is just a one
> > time
> > > > > cost.
> > > > > > > My
> > > > > > > > > > > concern is mainly about the long term burden. Assume we
> > have
> > > > > new
> > > > > > > source
> > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > connectors in
> > > > > old
> > > > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > > > interface,
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > > Pulsar connector might be deprecated almost immediately
> > after
> > > > > > > checked
> > > > > > > > > in,
> > > > > > > > > > > but we may still have to maintain two code bases. For the
> > > > > > existing
> > > > > > > > > > > connectors, we have to do that anyways. But it would be
> > good
> > > > to
> > > > > > > avoid
> > > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all!
> > > > > > > > > > > >
> > > > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > > > connector.
> > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > >
> > > > > > > > > > > > ## Contribute to Flink or maintain as a community
> > package
> > > > > > > > > > > >
> > > > > > > > > > > > Looks like the discussion is more going towards
> > > > > contribution. I
> > > > > > > think
> > > > > > > > > > > that
> > > > > > > > > > > > is good, especially if we think that we want to build a
> > > > > > similarly
> > > > > > > > > deep
> > > > > > > > > > > > integration with Pulsar as we have for example with
> > Kafka.
> > > > > The
> > > > > > > > > > connector
> > > > > > > > > > > > already looks like a more thorough connector than many
> > > > others
> > > > > > we
> > > > > > > have
> > > > > > > > > > in
> > > > > > > > > > > > the repository.
> > > > > > > > > > > >
> > > > > > > > > > > > With either a repo split, or the new build system, I
> > hope
> > > > > that
> > > > > > > the
> > > > > > > > > > build
> > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > >
> > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > >
> > > > > > > > > > > > Becket offered some help already, I can also help a
> > bit. I
> > > > > hope
> > > > > > > that
> > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > >
> > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > >
> > > > > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > > > > progress, but
> > > > > > > > > > > will
> > > > > > > > > > > > take 2 more months, I would estimate, before it is
> > fully
> > > > > > > available.
> > > > > > > > > > > >
> > > > > > > > > > > > If we want to be on the safe side with the
> > contribution, we
> > > > > > > should
> > > > > > > > > > > > contribute the source sooner and adjust it later. That
> > > > would
> > > > > > also
> > > > > > > > > help
> > > > > > > > > > us
> > > > > > > > > > > > in case things get crazy towards the 1.10 feature
> > freeze
> > > > and
> > > > > it
> > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > What would be the overhead of contributing now? Given
> > that
> > > > > the
> > > > > > > code
> > > > > > > > > is
> > > > > > > > > > > > already there, it looks like it would be only review
> > > > > capacity,
> > > > > > > right?
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Stephan
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your attention and the promotion of this
> > work.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > > > specific
> > > > > > > > > > > discussions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > > > consensus.
> > > > > > > > > > Therefore,
> > > > > > > > > > > > > I will explain all the functionalities of the
> > existing
> > > > > > > connector in
> > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > > > continue
> > > > > > our
> > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Yijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> > connector
> > > > in
> > > > > > > 1.10,
> > > > > > > > > how
> > > > > > > > > > > > about
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink
> > main
> > > > > repo
> > > > > > > as it
> > > > > > > > > > is a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away as
> > there
> > > > is
> > > > > > no
> > > > > > > > > change
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is
> > going to
> > > > > be
> > > > > > > code
> > > > > > > > > > > freeze
> > > > > > > > > > > > in
> > > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > > development
> > > > > > and
> > > > > > > > > > review
> > > > > > > > > > > of
> > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late
> > Oct.
> > > > > > There
> > > > > > > are
> > > > > > > > > > > still
> > > > > > > > > > > > 7
> > > > > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27
> > is
> > > > not
> > > > > > > ready
> > > > > > > > > by
> > > > > > > > > > > late
> > > > > > > > > > > > > > Oct, we can review and check in Pulsar connector
> > with
> > > > the
> > > > > > > > > existing
> > > > > > > > > > > > source
> > > > > > > > > > > > > > interface. This means we will have Pulsar
> > connector in
> > > > > > Flink
> > > > > > > > > 1.10,
> > > > > > > > > > > > either
> > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Because we are going to have Pulsar sink and source
> > > > > checked
> > > > > > > in
> > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > it might make sense to have two FLIPs, one for
> > Pulsar
> > > > > sink
> > > > > > > and
> > > > > > > > > > > another
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > Pulsar source. And we can start the work on Pulsar
> > sink
> > > > > > right
> > > > > > > > > away.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What's the take from Flink community? Shall we
> > wait
> > > > for
> > > > > > > FLIP-27
> > > > > > > > > > or
> > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > proceed to next steps? And what the next steps
> > are?
> > > > :-)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think having a Pulsar connector in Flink can
> > be a
> > > > > > good
> > > > > > > > > mutual
> > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Another perspective is that Pulsar connector
> > is the
> > > > > 1st
> > > > > > > > > > streaming
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > management
> > > > > system
> > > > > > > and
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > It'll be cool to see how the integration turns
> > out
> > > > > and
> > > > > > > > > whether
> > > > > > > > > > we
> > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> > currently in
> > > > > > > Beta, to
> > > > > > > > > > > cater
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> > merging
> > > > > > > Pulsar
> > > > > > > > > > > > connector
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g.
> > maybe
> > > > > one
> > > > > > > for
> > > > > > > > > > basic
> > > > > > > > > > > > > > > > source/sink functionalities and another for
> > schema
> > > > > and
> > > > > > > > > catalog
> > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27.
> > But I
> > > > > > don't
> > > > > > > > > think
> > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > should be a blocker in cases where it cannot
> > make
> > > > its
> > > > > > way
> > > > > > > > > into
> > > > > > > > > > > 1.10
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > > > committers
> > > > > > to
> > > > > > > > > > review
> > > > > > > > > > > or
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > interfaces.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > You are right. It all depends on when the new
> > > > > source
> > > > > > > > > > interface
> > > > > > > > > > > is
> > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > be ready. Personally I think it would be
> > there in
> > > > > > > about a
> > > > > > > > > > month
> > > > > > > > > > > > or
> > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > I could be too optimistic. It would also be
> > good
> > > > to
> > > > > > > hear
> > > > > > > > > what
> > > > > > > > > > > do
> > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > and Stephan think as they are also involved
> > in
> > > > > > FLIP-27.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > In general I think we should have Pulsar
> > > > connector
> > > > > in
> > > > > > > Flink
> > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > preferably with the new source interface. We
> > can
> > > > > also
> > > > > > > check
> > > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > with old source interface, but I suspect few
> > > > users
> > > > > > > will use
> > > > > > > > > > it
> > > > > > > > > > > > > before
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > > > reasonable
> > > > > > > to
> > > > > > > > > > wait a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > > > > interface. As
> > > > > > > > > > long
> > > > > > > > > > > > as
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > > > > doesn't
> > > > > > > seem
> > > > > > > > > to
> > > > > > > > > > > hurt
> > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann
> > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm wondering what the problem would be if
> > we
> > > > > > > committed
> > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > connector before the new source interface
> > is
> > > > > ready.
> > > > > > > If I
> > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > correctly, then we need to support the old
> > > > source
> > > > > > > > > interface
> > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > existing connectors. By checking it in
> > early I
> > > > > > could
> > > > > > > see
> > > > > > > > > > the
> > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > our users could start using the connector
> > > > > earlier.
> > > > > > > > > > Moreover,
> > > > > > > > > > > it
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > prevent that the Pulsar integration is
> > being
> > > > > > delayed
> > > > > > > in
> > > > > > > > > > case
> > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > source interface should be delayed. The
> > only
> > > > > > > downside I
> > > > > > > > > see
> > > > > > > > > > > is
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > review effort and potential fixes which
> > might
> > > > be
> > > > > > > > > irrelevant
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > source interface implementation. I guess it
> > > > > mainly
> > > > > > > > > depends
> > > > > > > > > > on
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > we are when the new source interface will
> > be
> > > > > ready.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > > > > Although
> > > > > > > the
> > > > > > > > > > FLIP
> > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > > > > time, a
> > > > > > > few
> > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > contributors in Flink community were
> > actually
> > > > > > > > > prototyping
> > > > > > > > > > > the
> > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > We have made some good progress there but
> > > > want
> > > > > to
> > > > > > > > > update
> > > > > > > > > > > the
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > after the entire thing is verified to
> > work in
> > > > > > case
> > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > minute surprise in the implementation. I
> > > > don't
> > > > > > > have an
> > > > > > > > > > > exact
> > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > guess it is going to be within a month
> > or so.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I am happy to review the current Flink
> > Pulsar
> > > > > > > connector
> > > > > > > > > > and
> > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to
> > avoid the
> > > > > > case
> > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar connector with some review
> > efforts and
> > > > > > > shortly
> > > > > > > > > > after
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie
> > Shen <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > > suggestions!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> > connector
> > > > has
> > > > > > > always
> > > > > > > > > > been
> > > > > > > > > > > to
> > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > users with the latest features of both
> > > > > systems
> > > > > > as
> > > > > > > > > soon
> > > > > > > > > > as
> > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > propose to contribute the connector to
> > > > Flink
> > > > > > and
> > > > > > > hope
> > > > > > > > > > to
> > > > > > > > > > > > get
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> > experts
> > > > > to
> > > > > > > ensure
> > > > > > > > > > the
> > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence
> > at
> > > > the
> > > > > > > > > beginning
> > > > > > > > > > of
> > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > the connector implementation based on
> > Flink
> > > > > > 1.9;
> > > > > > > we
> > > > > > > > > > also
> > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > build a connector that supports both
> > batch
> > > > > and
> > > > > > > stream
> > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > However, it has been inactive for some
> > > > time,
> > > > > so
> > > > > > > we
> > > > > > > > > > > decided
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > a connector with most of the new
> > features,
> > > > > such
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > and the new catalog API first. We will
> > pay
> > > > > > > attention
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate
> > it
> > > > > with
> > > > > > > the
> > > > > > > > > > > > connector
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Regarding the test status of the
> > connector,
> > > > > we
> > > > > > > are
> > > > > > > > > > > > following
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > connectors' test in Flink repository
> > and
> > > > > aimed
> > > > > > to
> > > > > > > > > > provide
> > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> > > > hear
> > > > > > > > > > suggestions
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > supervision from the Flink community to
> > > > > improve
> > > > > > > the
> > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > performance of the connector
> > continuously.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie
> > Guo <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It seems to me that the main question
> > > > here
> > > > > is
> > > > > > > > > about -
> > > > > > > > > > > > "how
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > > > > integration
> > > > > > > > > is
> > > > > > > > > > > kind
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > here. I believe there can be many
> > ways to
> > > > > > > achieve
> > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > As part of the contribution, what we
> > are
> > > > > > > looking
> > > > > > > > > for
> > > > > > > > > > > here
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > communities can build the
> > collaboration
> > > > > > > > > relationship
> > > > > > > > > > on
> > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > the integration between Pulsar and
> > Flink.
> > > > > > Even
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > try
> > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > community. We
> > > > > are
> > > > > > > still
> > > > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > > > experiences
> > > > > > > in
> > > > > > > > > > Flink
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > > > maintain
> > > > > > > and
> > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > integration
> > > > to
> > > > > > the
> > > > > > > > > users
> > > > > > > > > > > who
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > technologies, we need some help from
> > the
> > > > > > > experts
> > > > > > > > > from
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27
> > for a
> > > > > > while.
> > > > > > > > > > > Originally
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > of contributing the connectors back
> > after
> > > > > > > > > integrating
> > > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > > > > decided
> > > > > > > to
> > > > > > > > > > > initiate
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > as early as possible. Because we
> > believe
> > > > > > there
> > > > > > > are
> > > > > > > > > > more
> > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > > > > contribution,
> > > > > > > > > it
> > > > > > > > > > > can
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community understand more about
> > Pulsar
> > > > and
> > > > > > the
> > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > > > > verify
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> > > > Qin <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > contributing
> > > > > the
> > > > > > > > > Pulsar
> > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > > > connector
> > > > > > > with
> > > > > > > > > > > strong
> > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> > > > happy
> > > > > > the
> > > > > > > > > > > shepherd
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> > provide
> > > > > > some
> > > > > > > > > > context
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > > > > connector has
> > > > > > > > > > hit
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > With more and more connectors
> > coming
> > > > into
> > > > > > > Flink
> > > > > > > > > > repo,
> > > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > problems such as long build and
> > testing
> > > > > > > time. To
> > > > > > > > > > > > address
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > > > > separate
> > > > > > > > > > > repository.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > on hold due to potential solution
> > to
> > > > > > shorten
> > > > > > > the
> > > > > > > > > > > build
> > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to
> > stay as
> > > > > > > ecosystem
> > > > > > > > > > > > project
> > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > > functionality
> > > > > > and
> > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> > Ecosystem
> > > > > > > project
> > > > > > > > > > > website
> > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > through some final approval
> > process.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it would
> > be
> > > > > great
> > > > > > to
> > > > > > > > > first
> > > > > > > > > > > see
> > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > > > project
> > > > > > with
> > > > > > > > > great
> > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > > > connector
> > > > > > > is
> > > > > > > > > > tested
> > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > we can learn something to maintain
> > it
> > > > as
> > > > > an
> > > > > > > > > > ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > > > quality
> > > > > > as
> > > > > > > an
> > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt
> > it
> > > > > into
> > > > > > > the
> > > > > > > > > main
> > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> > FLIP-27
> > > > > > where
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > architecture and
> > > > > > > > > interface.
> > > > > > > > > > > This
> > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing
> > wise, if
> > > > > we
> > > > > > > are
> > > > > > > > > > going
> > > > > > > > > > > to
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> > wondering
> > > > if
> > > > > > we
> > > > > > > > > should
> > > > > > > > > > > > hold
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to
> > the
> > > > new
> > > > > > > > > interface
> > > > > > > > > > > to
> > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> > Chesnay
> > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may
> > end up
> > > > > > > repeating
> > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > > > > contributing a
> > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > of which failed because no
> > committer
> > > > > was
> > > > > > > > > getting
> > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > > > discussion
> > > > > > > > > thread
> > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > beforehand and getting several
> > +1's
> > > > > from
> > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > We should really make sure that
> > if we
> > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > contribution it will actually
> > get the
> > > > > > > attention
> > > > > > > > > > it
> > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> > recommend
> > > > > > > maintaining
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from
> > the
> > > > > > > > > documentation
> > > > > > > > > > > to
> > > > > > > > > > > > > give
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> > sharing
> > > > > > > artifacts
> > > > > > > > > > among
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?), this
> > may
> > > > > be a
> > > > > > > > > better
> > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> > Rohrmann
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > > > > discussion
> > > > > > > > > > Yijie.
> > > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> > valuable
> > > > > > > addition
> > > > > > > > > > since
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> > further
> > > > > > expand
> > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > from a project perspective it
> > makes
> > > > > > > sense for
> > > > > > > > > > me
> > > > > > > > > > > to
> > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > My main concern/question is
> > how can
> > > > > the
> > > > > > > Flink
> > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in the
> > past
> > > > > > that
> > > > > > > > > > > connectors
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > actively developed components
> > > > because
> > > > > > > they
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> > > > Given
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> > improving
> > > > > and
> > > > > > > > > > evolving
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> > > > this.
> > > > > > > Hence,
> > > > > > > > > +1
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> > > > Sijie
> > > > > > Guo
> > > > > > > <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver
> > behind
> > > > > > > > > FLINK-9641
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > > > > created
> > > > > > > for
> > > > > > > > > > > bringing
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > integration
> > > > was
> > > > > > done
> > > > > > > > > with
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago and
> > we
> > > > > ended
> > > > > > > up
> > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> > > > Flink
> > > > > > to
> > > > > > > > > > process
> > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > >
> > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty
> > simple
> > > > and
> > > > > > > there
> > > > > > > > > is
> > > > > > > > > > no
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have
> > made a
> > > > lot
> > > > > > of
> > > > > > > > > > changes
> > > > > > > > > > > in
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > first-class
> > > > > > > citizen in
> > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > > > processing
> > > > > > > > > Pulsar
> > > > > > > > > > > > event
> > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > > > integrate
> > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > > > connectors
> > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog
> > API as
> > > > > the
> > > > > > > > > > > first-class
> > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > >> integration. With that being
> > said,
> > > > > in
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a
> > flink
> > > > > > > catalog
> > > > > > > > > and
> > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> This is an example about how
> > to
> > > > use
> > > > > > > Pulsar
> > > > > > > > > as
> > > > > > > > > > a
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog
> > post
> > > > > > > > > explaining
> > > > > > > > > > > why
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and
> > what
> > > > > are
> > > > > > > the
> > > > > > > > > > > changes
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just
> > a
> > > > > simple
> > > > > > > data
> > > > > > > > > > sink
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> > integrated
> > > > > > > streaming
> > > > > > > > > > > data
> > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > schema/catalog
> > > > > > and
> > > > > > > > > > state).
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > > > > streaming
> > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > > > > processing.
> > > > > > > > > > Since
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration
> > to
> > > > > Flink
> > > > > > > here,
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve
> > the
> > > > > > > integration
> > > > > > > > > > > with
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > > > > background
> > > > > > > > > about
> > > > > > > > > > > the
> > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know what
> > are
> > > > > your
> > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54
> > AM
> > > > Yun
> > > > > > > Tang <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> > > > more
> > > > > > and
> > > > > > > more
> > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > > > > contribute to
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > discussion,
> > > > > would
> > > > > > > you
> > > > > > > > > > please
> > > > > > > > > > > > > give
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > > > > thread
> > > > > > to
> > > > > > > > > > current
> > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > > connector?
> > > > > > > Will the
> > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> > different
> > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3,
> > 2019
> > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > > > > Pulsar
> > > > > > > Flink
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > > > discussion
> > > > > > of
> > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to
> > Apache
> > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > > multi-tenant,
> > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> > Pulsar
> > > > > > > includes
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > > > clusters
> > > > > > > in a
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > > > messages
> > > > > > > across
> > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> > seamless
> > > > > > > > > scalability
> > > > > > > > > > to
> > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> > delivery
> > > > > with
> > > > > > > > > > persistent
> > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > BookKeeper.
> > > > > > > Nowadays,
> > > > > > > > > > Pulsar
> > > > > > > > > > > > has
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector
> > we are
> > > > > > > planning
> > > > > > > > > to
> > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar
> > 2.4.0. The
> > > > > > main
> > > > > > > > > > features
> > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> > source
> > > > with
> > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > > > Pulsar
> > > > > > with
> > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > exactly-once
> > > > > as
> > > > > > > well
> > > > > > > > > > when
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > >>> transaction features ready
> > in its
> > > > > > 2.5.0
> > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table
> > API
> > > > > Type
> > > > > > > > > system
> > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > > > messages
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > help
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > > > Catalog
> > > > > > API
> > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> > tables in
> > > > > > > Table API
> > > > > > > > > > as
> > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

[Becket Qin]

Hi Stephan,

Thanks for the clarification. I completely agree with you and Thomas on the
process of adding connectors to Flink repo. However, I am wondering what is
the deprecation process? Given the main concern here was that we may have
to maintain two Pulsar connector code bases until the old one is removed
from the repo, it would be good to know how long we have to do that.

Thanks,

Jiangjie (Becket) Qin

On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]> wrote:

> Some quick thoughts on the connector contribution process. I basically
> reiterate here what Thomas mentioned in another thread about the Kinesis
> connector.
>
> For connectors, we should favor a low-overhead contribution process, and
> accept user code and changes more readily than in the core system.
> That is because connectors have both a big variety of scenarios they get
> used in (only through use and many small contributions do they become
> really useful over time) and at the same time, and committers do not use
> the connector themselves and usually cannot foresee too well what is
> needed.
>
> Further more, a missing connector (or connector feature) is often a bigger
> show stopper for users than a missing API or system feature.
>
> Along these lines of thougt, the conclusion would be to take the Pulsar
> connector now, focus the review on legal/dependencies/rough code style and
> conventions, label it as "beta" (in the sense of "new code" that is "not
> yet tested through longer use") and go ahead. And then evolve it quickly
> without putting formal blockers in the way, meaning also adding a new FLIP
> 27 version when it is there.
>
> Best,
> Stephan
>
>
>
> On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]> wrote:
>
> > Hi Yijie,
> >
> > Could you please follow the FLIP process to start a new FLIP [DISCUSSION]
> > thread in the mailing list?
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> >
> > I see two FLIP-69 discussion in the mailing list now. So there is a FLIP
> > number collision. Can you change the FLIP number to 72?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]> wrote:
> >
> > > Hi Yijie,
> > >
> > > Thanks for sharing the pulsar FLIP.
> > > Would you mind enabling comments/suggestions on the google doc link?
> This
> > > way the contributors from the community can comment on the doc.
> > >
> > > Best,
> > > Rong
> > >
> > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <[hidden email]>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I've drafted a FLIP that describes the current design of the Pulsar
> > > > connector:
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > >
> > > > Please take a look and let me know what you think.
> > > >
> > > > Thanks,
> > > > Yijie
> > > >
> > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Sorry for joining the discussion late and thanks Yijie & Sijie for
> > > > driving
> > > > > the discussion.
> > > > > I also think the Pulsar connector would be a very valuable addition
> > to
> > > > > Flink. I can also help out a bit on the review side :-)
> > > > >
> > > > > Regarding the timeline, I also share concerns with Becket on the
> > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > There's also another discussion just started by Stephan on dropping
> > > Kafka
> > > > > 9/10 support on next Flink release [1].  Although the situation is
> > > > somewhat
> > > > > different, and Kafka 9/10 connector has been in Flink for almost
> 3-4
> > > > years,
> > > > > based on the discussion I am not sure if a major version release
> is a
> > > > > requirement for removing old connector supports.
> > > > >
> > > > > I think there shouldn't be a blocker if we agree the old connector
> > will
> > > > be
> > > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > > stated,
> > > > it
> > > > > is easier to contribute the source sooner and adjust it later.
> > > > > We should also ensure we clearly communicate the message: for
> > example,
> > > > > putting an experimental flag on the pre-FLIP27 connector page of
> the
> > > > > website, documentations, etc. Any other thoughts?
> > > > >
> > > > > --
> > > > > Rong
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > >
> > > > >
> > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Technically speaking, removing the old connector code is a
> > backwards
> > > > > > incompatible change which requires a major version bump, i.e.
> Flink
> > > > 2.x.
> > > > > > Given that we don't have a clear plan on when to have the next
> > major
> > > > > > version release, it seems unclear how long the old connector code
> > > will
> > > > be
> > > > > > there if we check it in right now. Or will we remove the old
> > > connector
> > > > > > without a major version bump? In any case, it sounds not quite
> user
> > > > > > friendly to the those who might use the old Pulsar connector. I
> am
> > > not
> > > > sure
> > > > > > if it is worth these potential problems in order to have the
> Pulsar
> > > > source
> > > > > > connector checked in one or two months earlier.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > > Agreed, if we check in the old code, we should make it clear
> that
> > > it
> > > > will
> > > > > > > be removed as soon as the FLIP-27 based version of the
> connector
> > is
> > > > > > there.
> > > > > > > We should not commit to maintaining the old version, that would
> > be
> > > > indeed
> > > > > > > too much overhead.
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> [hidden email]
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Stephan,
> > > > > > > >
> > > > > > > > Thanks for the volunteering to help.
> > > > > > > >
> > > > > > > > Yes, the overhead would just be review capacity. In fact, I
> am
> > > not
> > > > > > > worrying
> > > > > > > > too much about the review capacity. That is just a one time
> > cost.
> > > > My
> > > > > > > > concern is mainly about the long term burden. Assume we have
> > new
> > > > source
> > > > > > > > interface ready in 1.10 with newly added Pulsar connectors in
> > old
> > > > > > > > interface. Later on if we migrate Pulsar to new source
> > interface,
> > > > the
> > > > > > old
> > > > > > > > Pulsar connector might be deprecated almost immediately after
> > > > checked
> > > > > > in,
> > > > > > > > but we may still have to maintain two code bases. For the
> > > existing
> > > > > > > > connectors, we have to do that anyways. But it would be good
> to
> > > > avoid
> > > > > > > > introducing a new connector with the same problem.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> [hidden email]
> > >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all!
> > > > > > > > >
> > > > > > > > > Nice to see this lively discussion about the Pulsar
> > connector.
> > > > > > > > > Some thoughts on the open questions:
> > > > > > > > >
> > > > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > > > >
> > > > > > > > > Looks like the discussion is more going towards
> > contribution. I
> > > > think
> > > > > > > > that
> > > > > > > > > is good, especially if we think that we want to build a
> > > similarly
> > > > > > deep
> > > > > > > > > integration with Pulsar as we have for example with Kafka.
> > The
> > > > > > > connector
> > > > > > > > > already looks like a more thorough connector than many
> others
> > > we
> > > > have
> > > > > > > in
> > > > > > > > > the repository.
> > > > > > > > >
> > > > > > > > > With either a repo split, or the new build system, I hope
> > that
> > > > the
> > > > > > > build
> > > > > > > > > overhead is not a problem.
> > > > > > > > >
> > > > > > > > > ## Committer Support
> > > > > > > > >
> > > > > > > > > Becket offered some help already, I can also help a bit. I
> > hope
> > > > that
> > > > > > > > > between us, we can cover this.
> > > > > > > > >
> > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > >
> > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > progress, but
> > > > > > > > will
> > > > > > > > > take 2 more months, I would estimate, before it is fully
> > > > available.
> > > > > > > > >
> > > > > > > > > If we want to be on the safe side with the contribution, we
> > > > should
> > > > > > > > > contribute the source sooner and adjust it later. That
> would
> > > also
> > > > > > help
> > > > > > > us
> > > > > > > > > in case things get crazy towards the 1.10 feature freeze
> and
> > it
> > > > would
> > > > > > > be
> > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > What would be the overhead of contributing now? Given that
> > the
> > > > code
> > > > > > is
> > > > > > > > > already there, it looks like it would be only review
> > capacity,
> > > > right?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone!
> > > > > > > > > >
> > > > > > > > > > Thanks for your attention and the promotion of this work.
> > > > > > > > > >
> > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > specific
> > > > > > > > discussions.
> > > > > > > > > >
> > > > > > > > > > For FLIP-27, it seems that we have not reached a
> consensus.
> > > > > > > Therefore,
> > > > > > > > > > I will explain all the functionalities of the existing
> > > > connector in
> > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> continue
> > > our
> > > > > > > > > > discussions in FLIP.
> > > > > > > > > >
> > > > > > > > > > Thanks for your kind help.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yijie
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Sijie,
> > > > > > > > > > >
> > > > > > > > > > > If we agree that the goal is to have Pulsar connector
> in
> > > > 1.10,
> > > > > > how
> > > > > > > > > about
> > > > > > > > > > we
> > > > > > > > > > > do the following:
> > > > > > > > > > >
> > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main
> > repo
> > > > as it
> > > > > > > is a
> > > > > > > > > new
> > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > 1. Start to review the Pulsar sink right away as there
> is
> > > no
> > > > > > change
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > sink interface so far.
> > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going to
> > be
> > > > code
> > > > > > > > freeze
> > > > > > > > > in
> > > > > > > > > > > late Nov and let's say we give a month to the
> development
> > > and
> > > > > > > review
> > > > > > > > of
> > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late Oct.
> > > There
> > > > are
> > > > > > > > still
> > > > > > > > > 7
> > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is
> not
> > > > ready
> > > > > > by
> > > > > > > > late
> > > > > > > > > > > Oct, we can review and check in Pulsar connector with
> the
> > > > > > existing
> > > > > > > > > source
> > > > > > > > > > > interface. This means we will have Pulsar connector in
> > > Flink
> > > > > > 1.10,
> > > > > > > > > either
> > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > >
> > > > > > > > > > > Because we are going to have Pulsar sink and source
> > checked
> > > > in
> > > > > > > > > > separately,
> > > > > > > > > > > it might make sense to have two FLIPs, one for Pulsar
> > sink
> > > > and
> > > > > > > > another
> > > > > > > > > > for
> > > > > > > > > > > Pulsar source. And we can start the work on Pulsar sink
> > > right
> > > > > > away.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > >
> > > > > > > > > > > > What's the take from Flink community? Shall we wait
> for
> > > > FLIP-27
> > > > > > > or
> > > > > > > > > > shall we
> > > > > > > > > > > > proceed to next steps? And what the next steps are?
> :-)
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Sijie
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think having a Pulsar connector in Flink can be a
> > > good
> > > > > > mutual
> > > > > > > > > > benefit
> > > > > > > > > > > > to
> > > > > > > > > > > > > both communities.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Another perspective is that Pulsar connector is the
> > 1st
> > > > > > > streaming
> > > > > > > > > > > > connector
> > > > > > > > > > > > > that integrates with Flink's metadata management
> > system
> > > > and
> > > > > > > > Catalog
> > > > > > > > > > APIs.
> > > > > > > > > > > > > It'll be cool to see how the integration turns out
> > and
> > > > > > whether
> > > > > > > we
> > > > > > > > > > need to
> > > > > > > > > > > > > improve Flink Catalog stack, which are currently in
> > > > Beta, to
> > > > > > > > cater
> > > > > > > > > to
> > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of merging
> > > > Pulsar
> > > > > > > > > connector
> > > > > > > > > > into
> > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g. maybe
> > one
> > > > for
> > > > > > > basic
> > > > > > > > > > > > > source/sink functionalities and another for schema
> > and
> > > > > > catalog
> > > > > > > > > > > > integration,
> > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > >
> > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But I
> > > don't
> > > > > > think
> > > > > > > > > > FLIP-27
> > > > > > > > > > > > > should be a blocker in cases where it cannot make
> its
> > > way
> > > > > > into
> > > > > > > > 1.10
> > > > > > > > > > or
> > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > committers
> > > to
> > > > > > > review
> > > > > > > > or
> > > > > > > > > > for
> > > > > > > > > > > > > Pulsar connector to fully adapt to new interfaces.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Bowen
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You are right. It all depends on when the new
> > source
> > > > > > > interface
> > > > > > > > is
> > > > > > > > > > going
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > be ready. Personally I think it would be there in
> > > > about a
> > > > > > > month
> > > > > > > > > or
> > > > > > > > > > so.
> > > > > > > > > > > > > But
> > > > > > > > > > > > > > I could be too optimistic. It would also be good
> to
> > > > hear
> > > > > > what
> > > > > > > > do
> > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > and Stephan think as they are also involved in
> > > FLIP-27.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In general I think we should have Pulsar
> connector
> > in
> > > > Flink
> > > > > > > > 1.10,
> > > > > > > > > > > > > > preferably with the new source interface. We can
> > also
> > > > check
> > > > > > > it
> > > > > > > > in
> > > > > > > > > > right
> > > > > > > > > > > > > now
> > > > > > > > > > > > > > with old source interface, but I suspect few
> users
> > > > will use
> > > > > > > it
> > > > > > > > > > before
> > > > > > > > > > > > the
> > > > > > > > > > > > > > next official release. Therefore, it seems
> > reasonable
> > > > to
> > > > > > > wait a
> > > > > > > > > > little
> > > > > > > > > > > > > bit
> > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > interface. As
> > > > > > > long
> > > > > > > > > as
> > > > > > > > > > we
> > > > > > > > > > > > > make
> > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > doesn't
> > > > seem
> > > > > > to
> > > > > > > > hurt
> > > > > > > > > > much.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm wondering what the problem would be if we
> > > > committed
> > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > connector before the new source interface is
> > ready.
> > > > If I
> > > > > > > > > > understood
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > correctly, then we need to support the old
> source
> > > > > > interface
> > > > > > > > > > anyway
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > existing connectors. By checking it in early I
> > > could
> > > > see
> > > > > > > the
> > > > > > > > > > benefit
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > our users could start using the connector
> > earlier.
> > > > > > > Moreover,
> > > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > > > > > > prevent that the Pulsar integration is being
> > > delayed
> > > > in
> > > > > > > case
> > > > > > > > > > that the
> > > > > > > > > > > > > > > source interface should be delayed. The only
> > > > downside I
> > > > > > see
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > review effort and potential fixes which might
> be
> > > > > > irrelevant
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > > source interface implementation. I guess it
> > mainly
> > > > > > depends
> > > > > > > on
> > > > > > > > > how
> > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > we are when the new source interface will be
> > ready.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > Although
> > > > the
> > > > > > > FLIP
> > > > > > > > > > wiki
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > time, a
> > > > few
> > > > > > > > > > committer /
> > > > > > > > > > > > > > > > contributors in Flink community were actually
> > > > > > prototyping
> > > > > > > > the
> > > > > > > > > > > > entire
> > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > We have made some good progress there but
> want
> > to
> > > > > > update
> > > > > > > > the
> > > > > > > > > > FLIP
> > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > after the entire thing is verified to work in
> > > case
> > > > > > there
> > > > > > > > are
> > > > > > > > > > some
> > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > minute surprise in the implementation. I
> don't
> > > > have an
> > > > > > > > exact
> > > > > > > > > > ETA
> > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > guess it is going to be within a month or so.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am happy to review the current Flink Pulsar
> > > > connector
> > > > > > > and
> > > > > > > > > > see if
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid the
> > > case
> > > > that
> > > > > > > we
> > > > > > > > > > checked
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar connector with some review efforts and
> > > > shortly
> > > > > > > after
> > > > > > > > > > that
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for all the feedback and
> suggestions!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As Sijie said, the goal of the connector
> has
> > > > always
> > > > > > > been
> > > > > > > > to
> > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > users with the latest features of both
> > systems
> > > as
> > > > > > soon
> > > > > > > as
> > > > > > > > > > > > possible.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > propose to contribute the connector to
> Flink
> > > and
> > > > hope
> > > > > > > to
> > > > > > > > > get
> > > > > > > > > > more
> > > > > > > > > > > > > > > > > suggestions and feedback from Flink experts
> > to
> > > > ensure
> > > > > > > the
> > > > > > > > > > high
> > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at
> the
> > > > > > beginning
> > > > > > > of
> > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > the connector implementation based on Flink
> > > 1.9;
> > > > we
> > > > > > > also
> > > > > > > > > > wanted
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > build a connector that supports both batch
> > and
> > > > stream
> > > > > > > > > > computing
> > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > However, it has been inactive for some
> time,
> > so
> > > > we
> > > > > > > > decided
> > > > > > > > > to
> > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > a connector with most of the new features,
> > such
> > > > as
> > > > > > the
> > > > > > > > new
> > > > > > > > > > type
> > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > and the new catalog API first. We will pay
> > > > attention
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it
> > with
> > > > the
> > > > > > > > > connector
> > > > > > > > > > as
> > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regarding the test status of the connector,
> > we
> > > > are
> > > > > > > > > following
> > > > > > > > > > the
> > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > connectors' test in Flink repository and
> > aimed
> > > to
> > > > > > > provide
> > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> hear
> > > > > > > suggestions
> > > > > > > > > and
> > > > > > > > > > > > > > > > > supervision from the Flink community to
> > improve
> > > > the
> > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > performance of the connector continuously.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > feedback.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It seems to me that the main question
> here
> > is
> > > > > > about -
> > > > > > > > > "how
> > > > > > > > > > can
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > integration
> > > > > > is
> > > > > > > > kind
> > > > > > > > > > of
> > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > here. I believe there can be many ways to
> > > > achieve
> > > > > > it.
> > > > > > > > > > > > > > > > > > As part of the contribution, what we are
> > > > looking
> > > > > > for
> > > > > > > > here
> > > > > > > > > > is
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > communities can build the collaboration
> > > > > > relationship
> > > > > > > on
> > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > the integration between Pulsar and Flink.
> > > Even
> > > > we
> > > > > > can
> > > > > > > > try
> > > > > > > > > > our
> > > > > > > > > > > > > best
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > up all the updates in Flink community. We
> > are
> > > > still
> > > > > > > > > > > > > > > > > > facing the fact that we have less
> > experiences
> > > > in
> > > > > > > Flink
> > > > > > > > > than
> > > > > > > > > > > > folks
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community. In order to make sure we
> > maintain
> > > > and
> > > > > > > > deliver
> > > > > > > > > > > > > > > > > > a high-quality pulsar-flink integration
> to
> > > the
> > > > > > users
> > > > > > > > who
> > > > > > > > > > use
> > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > technologies, we need some help from the
> > > > experts
> > > > > > from
> > > > > > > > > Flink
> > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> > > while.
> > > > > > > > Originally
> > > > > > > > > > we
> > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > of contributing the connectors back after
> > > > > > integrating
> > > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > decided
> > > > to
> > > > > > > > initiate
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > as early as possible. Because we believe
> > > there
> > > > are
> > > > > > > more
> > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > contribution,
> > > > > > it
> > > > > > > > can
> > > > > > > > > > help
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community understand more about Pulsar
> and
> > > the
> > > > > > > > potential
> > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > verify
> > > > the
> > > > > > new
> > > > > > > > > > connector
> > > > > > > > > > > > > API
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> Qin <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for the interest in contributing
> > the
> > > > > > Pulsar
> > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > connector
> > > > with
> > > > > > > > strong
> > > > > > > > > > > > support
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> happy
> > > the
> > > > > > > > shepherd
> > > > > > > > > > this
> > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to provide
> > > some
> > > > > > > context
> > > > > > > > > and
> > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > connector has
> > > > > > > hit
> > > > > > > > > the
> > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > With more and more connectors coming
> into
> > > > Flink
> > > > > > > repo,
> > > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > problems such as long build and testing
> > > > time. To
> > > > > > > > > address
> > > > > > > > > > this
> > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > separate
> > > > > > > > repository.
> > > > > > > > > > This
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > on hold due to potential solution to
> > > shorten
> > > > the
> > > > > > > > build
> > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay as
> > > > ecosystem
> > > > > > > > > project
> > > > > > > > > > > > while
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > to provide good support for
> functionality
> > > and
> > > > > > > > > > compatibility
> > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > has driven to create a Flink Ecosystem
> > > > project
> > > > > > > > website
> > > > > > > > > > and it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Given the above efforts, it would be
> > great
> > > to
> > > > > > first
> > > > > > > > see
> > > > > > > > > > if we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> project
> > > with
> > > > > > great
> > > > > > > > > > support.
> > > > > > > > > > > > It
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > connector
> > > > is
> > > > > > > tested
> > > > > > > > > > > > currently
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > we can learn something to maintain it
> as
> > an
> > > > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> quality
> > > as
> > > > an
> > > > > > > > > ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it
> > into
> > > > the
> > > > > > main
> > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is FLIP-27
> > > where
> > > > we
> > > > > > are
> > > > > > > > > > making
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Flink source connector architecture and
> > > > > > interface.
> > > > > > > > This
> > > > > > > > > > > > change
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise, if
> > we
> > > > are
> > > > > > > going
> > > > > > > > to
> > > > > > > > > > have
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > connector in main repo, I am wondering
> if
> > > we
> > > > > > should
> > > > > > > > > hold
> > > > > > > > > > a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the
> new
> > > > > > interface
> > > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM Chesnay
> > > > Schepler <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm quite worried that we may end up
> > > > repeating
> > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > contributing a
> > > > > > > > > pulsar
> > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > of which failed because no committer
> > was
> > > > > > getting
> > > > > > > > > > involved,
> > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > discussion
> > > > > > thread
> > > > > > > > > > about the
> > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > beforehand and getting several +1's
> > from
> > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > We should really make sure that if we
> > > > > > > > welcome/approve
> > > > > > > > > > such
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > contribution it will actually get the
> > > > attention
> > > > > > > it
> > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > > > maintaining
> > > > > > > the
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > > > documentation
> > > > > > > > to
> > > > > > > > > > give
> > > > > > > > > > > > it
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > > > artifacts
> > > > > > > among
> > > > > > > > > the
> > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > the state of that anyway?), this may
> > be a
> > > > > > better
> > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann
> > wrote:
> > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > discussion
> > > > > > > Yijie.
> > > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > connector would be a very valuable
> > > > addition
> > > > > > > since
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > more popular and it would further
> > > expand
> > > > > > > Flink's
> > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > from a project perspective it makes
> > > > sense for
> > > > > > > me
> > > > > > > > to
> > > > > > > > > > place
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > My main concern/question is how can
> > the
> > > > Flink
> > > > > > > > > > community
> > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector? We have seen in the past
> > > that
> > > > > > > > connectors
> > > > > > > > > > are
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > actively developed components
> because
> > > > they
> > > > > > need
> > > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > > kept
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> Given
> > > > that
> > > > > > the
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > to help with maintaining, improving
> > and
> > > > > > > evolving
> > > > > > > > > the
> > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> this.
> > > > Hence,
> > > > > > +1
> > > > > > > > for
> > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> Sijie
> > > Guo
> > > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Since I was the main driver behind
> > > > > > FLINK-9641
> > > > > > > > and
> > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > created
> > > > for
> > > > > > > > bringing
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > >> sink for Flink. The integration
> was
> > > done
> > > > > > with
> > > > > > > > > Flink
> > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > >> requests about a year ago and we
> > ended
> > > > up
> > > > > > > > > > maintaining
> > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> Flink
> > > to
> > > > > > > process
> > > > > > > > > > event
> > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > >
> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple
> and
> > > > there
> > > > > > is
> > > > > > > no
> > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> In the past year, we have made a
> lot
> > > of
> > > > > > > changes
> > > > > > > > in
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > > > citizen in
> > > > > > > > > Pulsar.
> > > > > > > > > > We
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > processing
> > > > > > Pulsar
> > > > > > > > > event
> > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > integrate
> > > > with
> > > > > > > Flink
> > > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > connectors
> > > > from
> > > > > > > the
> > > > > > > > > > ground
> > > > > > > > > > > > up
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > >> bring table API and catalog API as
> > the
> > > > > > > > first-class
> > > > > > > > > > > > citizen
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > >> integration. With that being said,
> > in
> > > > the
> > > > > > new
> > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a flink
> > > > catalog
> > > > > > and
> > > > > > > > > > query /
> > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> This is an example about how to
> use
> > > > Pulsar
> > > > > > as
> > > > > > > a
> > > > > > > > > > Flink
> > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog post
> > > > > > explaining
> > > > > > > > why
> > > > > > > > > we
> > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and what
> > are
> > > > the
> > > > > > > > changes
> > > > > > > > > we
> > > > > > > > > > > > made
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a
> > simple
> > > > data
> > > > > > > sink
> > > > > > > > > or
> > > > > > > > > > > > source
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > >> actually can be a fully integrated
> > > > streaming
> > > > > > > > data
> > > > > > > > > > > > storage
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> schema/catalog
> > > and
> > > > > > > state).
> > > > > > > > > The
> > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > streaming
> > > > > > > > warehouse
> > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > processing.
> > > > > > > Since
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to
> > Flink
> > > > here,
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > also
> > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > > > integration
> > > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > background
> > > > > > about
> > > > > > > > the
> > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > >> integration. Let me know what are
> > your
> > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM
> Yun
> > > > Tang <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> more
> > > and
> > > > more
> > > > > > > > > popular
> > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > contribute to
> > > > > > > > Flink
> > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Before any further discussion,
> > would
> > > > you
> > > > > > > please
> > > > > > > > > > give
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > thread
> > > to
> > > > > > > current
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> connector?
> > > > Will the
> > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > >>> ________________________________
> > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3, 2019
> > > 13:57
> > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > Pulsar
> > > > Flink
> > > > > > > > > > connector
> > > > > > > > > > > > back
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> discussion
> > > of
> > > > > > > > > contributing
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to Apache
> > > > Pulsar
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> multi-tenant,
> > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system. Pulsar
> > > > includes
> > > > > > > > > multiple
> > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > clusters
> > > > in a
> > > > > > > > Pulsar
> > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > messages
> > > > across
> > > > > > > > > > clusters,
> > > > > > > > > > > > > very
> > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency, seamless
> > > > > > scalability
> > > > > > > to
> > > > > > > > > > over a
> > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery
> > with
> > > > > > > persistent
> > > > > > > > > > message
> > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > > > Nowadays,
> > > > > > > Pulsar
> > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > connector
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we are
> > > > planning
> > > > > > to
> > > > > > > > > > > > contribute
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0. The
> > > main
> > > > > > > features
> > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source
> with
> > > > > > > > exactly-once
> > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> Pulsar
> > > with
> > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > >>> would update this to exactly-once
> > as
> > > > well
> > > > > > > when
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > >>> transaction features ready in its
> > > 2.5.0
> > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table API
> > Type
> > > > > > system
> > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > messages
> > > > with
> > > > > > the
> > > > > > > > > help
> > > > > > > > > > of
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> Catalog
> > > API
> > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables in
> > > > Table API
> > > > > > > as
> > > > > > > > > > well as
> > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>

[Fabian Hueske-2]

Hi,

You should have the necessary permissions.
Let me know if something doesn't work.

Cheers, Fabian

Am Do., 19. Sept. 2019 um 10:56 Uhr schrieb Yijie Shen <
[hidden email]>:

> Hi Fabian,
>
> It's   yijieshen
>
> Thanks for your help!
>
> On Thu, Sep 19, 2019 at 4:52 PM Fabian Hueske <[hidden email]> wrote:
> >
> > Hi Yijie,
> >
> > I can give you permission if you tell me your Confluence user name.
> >
> > Thanks, Fabian
> >
> > Am Do., 19. Sept. 2019 um 10:49 Uhr schrieb Yijie Shen <
> > [hidden email]>:
> >
> > > > > > Could you please follow the FLIP process to start a new FLIP
> > > [DISCUSSION]
> > > > > > thread in the mailing list?
> > >
> > > It seems that I don't have permission to create pages in FLIP.
> > >
> > >
> > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > Would you mind enabling comments/suggestions on the google doc
> > > link?
> > >
> > > Sorry, my bad, I've enabled comments for the doc.
> > > Does it work for now?
> > >
> > > On Thu, Sep 19, 2019 at 4:07 PM Fabian Hueske <[hidden email]>
> wrote:
> > > >
> > > > +1 to what Stephan (and Thomas) said.
> > > >
> > > > Am Do., 19. Sept. 2019 um 09:54 Uhr schrieb Stephan Ewen <
> > > [hidden email]>:
> > > >
> > > > > Some quick thoughts on the connector contribution process. I
> basically
> > > > > reiterate here what Thomas mentioned in another thread about the
> > > Kinesis
> > > > > connector.
> > > > >
> > > > > For connectors, we should favor a low-overhead contribution
> process,
> > > and
> > > > > accept user code and changes more readily than in the core system.
> > > > > That is because connectors have both a big variety of scenarios
> they
> > > get
> > > > > used in (only through use and many small contributions do they
> become
> > > > > really useful over time) and at the same time, and committers do
> not
> > > use
> > > > > the connector themselves and usually cannot foresee too well what
> is
> > > > > needed.
> > > > >
> > > > > Further more, a missing connector (or connector feature) is often a
> > > bigger
> > > > > show stopper for users than a missing API or system feature.
> > > > >
> > > > > Along these lines of thougt, the conclusion would be to take the
> Pulsar
> > > > > connector now, focus the review on legal/dependencies/rough code
> style
> > > and
> > > > > conventions, label it as "beta" (in the sense of "new code" that is
> > > "not
> > > > > yet tested through longer use") and go ahead. And then evolve it
> > > quickly
> > > > > without putting formal blockers in the way, meaning also adding a
> new
> > > FLIP
> > > > > 27 version when it is there.
> > > > >
> > > > > Best,
> > > > > Stephan
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hi Yijie,
> > > > > >
> > > > > > Could you please follow the FLIP process to start a new FLIP
> > > [DISCUSSION]
> > > > > > thread in the mailing list?
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > >
> > > > > > I see two FLIP-69 discussion in the mailing list now. So there
> is a
> > > FLIP
> > > > > > number collision. Can you change the FLIP number to 72?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > > Hi Yijie,
> > > > > > >
> > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > Would you mind enabling comments/suggestions on the google doc
> > > link?
> > > > > This
> > > > > > > way the contributors from the community can comment on the doc.
> > > > > > >
> > > > > > > Best,
> > > > > > > Rong
> > > > > > >
> > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > I've drafted a FLIP that describes the current design of the
> > > Pulsar
> > > > > > > > connector:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > >
> > > > > > > > Please take a look and let me know what you think.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yijie
> > > > > > > >
> > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> [hidden email]>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Sorry for joining the discussion late and thanks Yijie &
> Sijie
> > > for
> > > > > > > > driving
> > > > > > > > > the discussion.
> > > > > > > > > I also think the Pulsar connector would be a very valuable
> > > addition
> > > > > > to
> > > > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > > > >
> > > > > > > > > Regarding the timeline, I also share concerns with Becket
> on
> > > the
> > > > > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > > > > There's also another discussion just started by Stephan on
> > > dropping
> > > > > > > Kafka
> > > > > > > > > 9/10 support on next Flink release [1].  Although the
> > > situation is
> > > > > > > > somewhat
> > > > > > > > > different, and Kafka 9/10 connector has been in Flink for
> > > almost
> > > > > 3-4
> > > > > > > > years,
> > > > > > > > > based on the discussion I am not sure if a major version
> > > release
> > > > > is a
> > > > > > > > > requirement for removing old connector supports.
> > > > > > > > >
> > > > > > > > > I think there shouldn't be a blocker if we agree the old
> > > connector
> > > > > > will
> > > > > > > > be
> > > > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> > > Stephan
> > > > > > > stated,
> > > > > > > > it
> > > > > > > > > is easier to contribute the source sooner and adjust it
> later.
> > > > > > > > > We should also ensure we clearly communicate the message:
> for
> > > > > > example,
> > > > > > > > > putting an experimental flag on the pre-FLIP27 connector
> page
> > > of
> > > > > the
> > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rong
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Technically speaking, removing the old connector code is
> a
> > > > > > backwards
> > > > > > > > > > incompatible change which requires a major version bump,
> i.e.
> > > > > Flink
> > > > > > > > 2.x.
> > > > > > > > > > Given that we don't have a clear plan on when to have the
> > > next
> > > > > > major
> > > > > > > > > > version release, it seems unclear how long the old
> connector
> > > code
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > there if we check it in right now. Or will we remove the
> old
> > > > > > > connector
> > > > > > > > > > without a major version bump? In any case, it sounds not
> > > quite
> > > > > user
> > > > > > > > > > friendly to the those who might use the old Pulsar
> > > connector. I
> > > > > am
> > > > > > > not
> > > > > > > > sure
> > > > > > > > > > if it is worth these potential problems in order to have
> the
> > > > > Pulsar
> > > > > > > > source
> > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Agreed, if we check in the old code, we should make it
> > > clear
> > > > > that
> > > > > > > it
> > > > > > > > will
> > > > > > > > > > > be removed as soon as the FLIP-27 based version of the
> > > > > connector
> > > > > > is
> > > > > > > > > > there.
> > > > > > > > > > > We should not commit to maintaining the old version,
> that
> > > would
> > > > > > be
> > > > > > > > indeed
> > > > > > > > > > > too much overhead.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, the overhead would just be review capacity. In
> > > fact, I
> > > > > am
> > > > > > > not
> > > > > > > > > > > worrying
> > > > > > > > > > > > too much about the review capacity. That is just a
> one
> > > time
> > > > > > cost.
> > > > > > > > My
> > > > > > > > > > > > concern is mainly about the long term burden. Assume
> we
> > > have
> > > > > > new
> > > > > > > > source
> > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > > connectors in
> > > > > > old
> > > > > > > > > > > > interface. Later on if we migrate Pulsar to new
> source
> > > > > > interface,
> > > > > > > > the
> > > > > > > > > > old
> > > > > > > > > > > > Pulsar connector might be deprecated almost
> immediately
> > > after
> > > > > > > > checked
> > > > > > > > > > in,
> > > > > > > > > > > > but we may still have to maintain two code bases.
> For the
> > > > > > > existing
> > > > > > > > > > > > connectors, we have to do that anyways. But it would
> be
> > > good
> > > > > to
> > > > > > > > avoid
> > > > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > > > > connector.
> > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > >
> > > > > > > > > > > > > ## Contribute to Flink or maintain as a community
> > > package
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looks like the discussion is more going towards
> > > > > > contribution. I
> > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > is good, especially if we think that we want to
> build a
> > > > > > > similarly
> > > > > > > > > > deep
> > > > > > > > > > > > > integration with Pulsar as we have for example with
> > > Kafka.
> > > > > > The
> > > > > > > > > > > connector
> > > > > > > > > > > > > already looks like a more thorough connector than
> many
> > > > > others
> > > > > > > we
> > > > > > > > have
> > > > > > > > > > > in
> > > > > > > > > > > > > the repository.
> > > > > > > > > > > > >
> > > > > > > > > > > > > With either a repo split, or the new build system,
> I
> > > hope
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > build
> > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > >
> > > > > > > > > > > > > Becket offered some help already, I can also help a
> > > bit. I
> > > > > > hope
> > > > > > > > that
> > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > >
> > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some
> PoC-ing
> > > > > > > > progress, but
> > > > > > > > > > > > will
> > > > > > > > > > > > > take 2 more months, I would estimate, before it is
> > > fully
> > > > > > > > available.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we want to be on the safe side with the
> > > contribution, we
> > > > > > > > should
> > > > > > > > > > > > > contribute the source sooner and adjust it later.
> That
> > > > > would
> > > > > > > also
> > > > > > > > > > help
> > > > > > > > > > > us
> > > > > > > > > > > > > in case things get crazy towards the 1.10 feature
> > > freeze
> > > > > and
> > > > > > it
> > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > > What would be the overhead of contributing now?
> Given
> > > that
> > > > > > the
> > > > > > > > code
> > > > > > > > > > is
> > > > > > > > > > > > > already there, it looks like it would be only
> review
> > > > > > capacity,
> > > > > > > > right?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Stephan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your attention and the promotion of
> this
> > > work.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We will prepare a FLIP as soon as possible for
> more
> > > > > > specific
> > > > > > > > > > > > discussions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > > > > consensus.
> > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > I will explain all the functionalities of the
> > > existing
> > > > > > > > connector in
> > > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > > > > continue
> > > > > > > our
> > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> > > connector
> > > > > in
> > > > > > > > 1.10,
> > > > > > > > > > how
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to
> Flink
> > > main
> > > > > > repo
> > > > > > > > as it
> > > > > > > > > > > is a
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away
> as
> > > there
> > > > > is
> > > > > > > no
> > > > > > > > > > change
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is
> > > going to
> > > > > > be
> > > > > > > > code
> > > > > > > > > > > > freeze
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > > > development
> > > > > > > and
> > > > > > > > > > > review
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by
> late
> > > Oct.
> > > > > > > There
> > > > > > > > are
> > > > > > > > > > > > still
> > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > weeks. Personally I think it is doable. If
> FLIP-27
> > > is
> > > > > not
> > > > > > > > ready
> > > > > > > > > > by
> > > > > > > > > > > > late
> > > > > > > > > > > > > > > Oct, we can review and check in Pulsar
> connector
> > > with
> > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > interface. This means we will have Pulsar
> > > connector in
> > > > > > > Flink
> > > > > > > > > > 1.10,
> > > > > > > > > > > > > either
> > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Because we are going to have Pulsar sink and
> source
> > > > > > checked
> > > > > > > > in
> > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > it might make sense to have two FLIPs, one for
> > > Pulsar
> > > > > > sink
> > > > > > > > and
> > > > > > > > > > > > another
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > Pulsar source. And we can start the work on
> Pulsar
> > > sink
> > > > > > > right
> > > > > > > > > > away.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What's the take from Flink community? Shall
> we
> > > wait
> > > > > for
> > > > > > > > FLIP-27
> > > > > > > > > > > or
> > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > proceed to next steps? And what the next
> steps
> > > are?
> > > > > :-)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think having a Pulsar connector in Flink
> can
> > > be a
> > > > > > > good
> > > > > > > > > > mutual
> > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Another perspective is that Pulsar
> connector
> > > is the
> > > > > > 1st
> > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > > management
> > > > > > system
> > > > > > > > and
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > It'll be cool to see how the integration
> turns
> > > out
> > > > > > and
> > > > > > > > > > whether
> > > > > > > > > > > we
> > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> > > currently in
> > > > > > > > Beta, to
> > > > > > > > > > > > cater
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> > > merging
> > > > > > > > Pulsar
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs,
> e.g.
> > > maybe
> > > > > > one
> > > > > > > > for
> > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > source/sink functionalities and another for
> > > schema
> > > > > > and
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for
> FLIP-27.
> > > But I
> > > > > > > don't
> > > > > > > > > > think
> > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > should be a blocker in cases where it
> cannot
> > > make
> > > > > its
> > > > > > > way
> > > > > > > > > > into
> > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > > > > committers
> > > > > > > to
> > > > > > > > > > > review
> > > > > > > > > > > > or
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > > interfaces.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > You are right. It all depends on when
> the new
> > > > > > source
> > > > > > > > > > > interface
> > > > > > > > > > > > is
> > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > be ready. Personally I think it would be
> > > there in
> > > > > > > > about a
> > > > > > > > > > > month
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > I could be too optimistic. It would also
> be
> > > good
> > > > > to
> > > > > > > > hear
> > > > > > > > > > what
> > > > > > > > > > > > do
> > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > and Stephan think as they are also
> involved
> > > in
> > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > In general I think we should have Pulsar
> > > > > connector
> > > > > > in
> > > > > > > > Flink
> > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > preferably with the new source
> interface. We
> > > can
> > > > > > also
> > > > > > > > check
> > > > > > > > > > > it
> > > > > > > > > > > > in
> > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > with old source interface, but I suspect
> few
> > > > > users
> > > > > > > > will use
> > > > > > > > > > > it
> > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > next official release. Therefore, it
> seems
> > > > > > reasonable
> > > > > > > > to
> > > > > > > > > > > wait a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > to see whether we can jump to the new
> source
> > > > > > > > interface. As
> > > > > > > > > > > long
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little
> bit
> > > > > > doesn't
> > > > > > > > seem
> > > > > > > > > > to
> > > > > > > > > > > > hurt
> > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till
> Rohrmann
> > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm wondering what the problem would
> be if
> > > we
> > > > > > > > committed
> > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > connector before the new source
> interface
> > > is
> > > > > > ready.
> > > > > > > > If I
> > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > correctly, then we need to support the
> old
> > > > > source
> > > > > > > > > > interface
> > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > existing connectors. By checking it in
> > > early I
> > > > > > > could
> > > > > > > > see
> > > > > > > > > > > the
> > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > our users could start using the
> connector
> > > > > > earlier.
> > > > > > > > > > > Moreover,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration is
> > > being
> > > > > > > delayed
> > > > > > > > in
> > > > > > > > > > > case
> > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > source interface should be delayed. The
> > > only
> > > > > > > > downside I
> > > > > > > > > > see
> > > > > > > > > > > > is
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > review effort and potential fixes which
> > > might
> > > > > be
> > > > > > > > > > irrelevant
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > source interface implementation. I
> guess it
> > > > > > mainly
> > > > > > > > > > depends
> > > > > > > > > > > on
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > we are when the new source interface
> will
> > > be
> > > > > > ready.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket
> Qin <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Just want to have some update on
> FLIP-27.
> > > > > > > Although
> > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > discussion thread has been quiet for
> some
> > > > > > time, a
> > > > > > > > few
> > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > contributors in Flink community were
> > > actually
> > > > > > > > > > prototyping
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > We have made some good progress
> there but
> > > > > want
> > > > > > to
> > > > > > > > > > update
> > > > > > > > > > > > the
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > after the entire thing is verified to
> > > work in
> > > > > > > case
> > > > > > > > > > there
> > > > > > > > > > > > are
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > minute surprise in the
> implementation. I
> > > > > don't
> > > > > > > > have an
> > > > > > > > > > > > exact
> > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > guess it is going to be within a
> month
> > > or so.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I am happy to review the current
> Flink
> > > Pulsar
> > > > > > > > connector
> > > > > > > > > > > and
> > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to
> > > avoid the
> > > > > > > case
> > > > > > > > that
> > > > > > > > > > > we
> > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Pulsar connector with some review
> > > efforts and
> > > > > > > > shortly
> > > > > > > > > > > after
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie
> > > Shen <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> > > connector
> > > > > has
> > > > > > > > always
> > > > > > > > > > > been
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > users with the latest features of
> both
> > > > > > systems
> > > > > > > as
> > > > > > > > > > soon
> > > > > > > > > > > as
> > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > propose to contribute the
> connector to
> > > > > Flink
> > > > > > > and
> > > > > > > > hope
> > > > > > > > > > > to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> > > experts
> > > > > > to
> > > > > > > > ensure
> > > > > > > > > > > the
> > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its
> existence
> > > at
> > > > > the
> > > > > > > > > > beginning
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > the connector implementation based
> on
> > > Flink
> > > > > > > 1.9;
> > > > > > > > we
> > > > > > > > > > > also
> > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > build a connector that supports
> both
> > > batch
> > > > > > and
> > > > > > > > stream
> > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > However, it has been inactive for
> some
> > > > > time,
> > > > > > so
> > > > > > > > we
> > > > > > > > > > > > decided
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > a connector with most of the new
> > > features,
> > > > > > such
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > and the new catalog API first. We
> will
> > > pay
> > > > > > > > attention
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> incorporate
> > > it
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Regarding the test status of the
> > > connector,
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > > following
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> repository
> > > and
> > > > > > aimed
> > > > > > > to
> > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > tests as we could. We are also
> happy to
> > > > > hear
> > > > > > > > > > > suggestions
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > supervision from the Flink
> community to
> > > > > > improve
> > > > > > > > the
> > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > performance of the connector
> > > continuously.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM
> Sijie
> > > Guo <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments
> and
> > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It seems to me that the main
> question
> > > > > here
> > > > > > is
> > > > > > > > > > about -
> > > > > > > > > > > > > "how
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community maintain the
> connector?".
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from
> myself.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to host
> this
> > > > > > > > integration
> > > > > > > > > > is
> > > > > > > > > > > > kind
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > here. I believe there can be many
> > > ways to
> > > > > > > > achieve
> > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > As part of the contribution,
> what we
> > > are
> > > > > > > > looking
> > > > > > > > > > for
> > > > > > > > > > > > here
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > communities can build the
> > > collaboration
> > > > > > > > > > relationship
> > > > > > > > > > > on
> > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > the integration between Pulsar
> and
> > > Flink.
> > > > > > > Even
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > try
> > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > > community. We
> > > > > > are
> > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > > > > experiences
> > > > > > > > in
> > > > > > > > > > > Flink
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community. In order to make sure
> we
> > > > > > maintain
> > > > > > > > and
> > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > > integration
> > > > > to
> > > > > > > the
> > > > > > > > > > users
> > > > > > > > > > > > who
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > technologies, we need some help
> from
> > > the
> > > > > > > > experts
> > > > > > > > > > from
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27
> > > for a
> > > > > > > while.
> > > > > > > > > > > > Originally
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > of contributing the connectors
> back
> > > after
> > > > > > > > > > integrating
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27.
> But we
> > > > > > decided
> > > > > > > > to
> > > > > > > > > > > > initiate
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > as early as possible. Because we
> > > believe
> > > > > > > there
> > > > > > > > are
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > it now rather than later. As
> part of
> > > > > > > > contribution,
> > > > > > > > > > it
> > > > > > > > > > > > can
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community understand more about
> > > Pulsar
> > > > > and
> > > > > > > the
> > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink
> community
> > > > > > verify
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog
> API).
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM
> Becket
> > > > > Qin <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > > contributing
> > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > In general, I think having
> Pulsar
> > > > > > connector
> > > > > > > > with
> > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So
> I am
> > > > > happy
> > > > > > > the
> > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> > > provide
> > > > > > > some
> > > > > > > > > > > context
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The current way Flink
> maintains its
> > > > > > > > connector has
> > > > > > > > > > > hit
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > With more and more connectors
> > > coming
> > > > > into
> > > > > > > > Flink
> > > > > > > > > > > repo,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > problems such as long build and
> > > testing
> > > > > > > > time. To
> > > > > > > > > > > > > address
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > have attempted to do the
> following:
> > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors
> into a
> > > > > > separate
> > > > > > > > > > > > repository.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > on hold due to potential
> solution
> > > to
> > > > > > > shorten
> > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to
> > > stay as
> > > > > > > > ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > > > functionality
> > > > > > > and
> > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> > > Ecosystem
> > > > > > > > project
> > > > > > > > > > > > website
> > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > through some final approval
> > > process.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it
> would
> > > be
> > > > > > great
> > > > > > > to
> > > > > > > > > > first
> > > > > > > > > > > > see
> > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an
> ecosystem
> > > > > project
> > > > > > > with
> > > > > > > > > > great
> > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink
> Pulsar
> > > > > > connector
> > > > > > > > is
> > > > > > > > > > > tested
> > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > we can learn something to
> maintain
> > > it
> > > > > as
> > > > > > an
> > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If
> the
> > > > > quality
> > > > > > > as
> > > > > > > > an
> > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well
> adopt
> > > it
> > > > > > into
> > > > > > > > the
> > > > > > > > > > main
> > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> > > FLIP-27
> > > > > > > where
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > > architecture and
> > > > > > > > > > interface.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing
> > > wise, if
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > going
> > > > > > > > > > > > to
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> > > wondering
> > > > > if
> > > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > > hold
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt
> to
> > > the
> > > > > new
> > > > > > > > > > interface
> > > > > > > > > > > > to
> > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> > > Chesnay
> > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may
> > > end up
> > > > > > > > repeating
> > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There were already 2
> attempts at
> > > > > > > > contributing a
> > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > of which failed because no
> > > committer
> > > > > > was
> > > > > > > > > > getting
> > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > contributor opening a
> dedicated
> > > > > > > discussion
> > > > > > > > > > thread
> > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting
> several
> > > +1's
> > > > > > from
> > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > We should really make sure
> that
> > > if we
> > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > contribution it will actually
> > > get the
> > > > > > > > attention
> > > > > > > > > > > it
> > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> > > recommend
> > > > > > > > maintaining
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it
> from
> > > the
> > > > > > > > > > documentation
> > > > > > > > > > > > to
> > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> > > sharing
> > > > > > > > artifacts
> > > > > > > > > > > among
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?),
> this
> > > may
> > > > > > be a
> > > > > > > > > > better
> > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> > > Rohrmann
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting
> this
> > > > > > > discussion
> > > > > > > > > > > Yijie.
> > > > > > > > > > > > I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> > > valuable
> > > > > > > > addition
> > > > > > > > > > > since
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> > > further
> > > > > > > expand
> > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > from a project perspective
> it
> > > makes
> > > > > > > > sense for
> > > > > > > > > > > me
> > > > > > > > > > > > to
> > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > My main concern/question is
> > > how can
> > > > > > the
> > > > > > > > Flink
> > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in
> the
> > > past
> > > > > > > that
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > actively developed
> components
> > > > > because
> > > > > > > > they
> > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > external system and with
> Flink.
> > > > > Given
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> > > improving
> > > > > > and
> > > > > > > > > > > evolving
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can
> achieve
> > > > > this.
> > > > > > > > Hence,
> > > > > > > > > > +1
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> 2:03 AM
> > > > > Sijie
> > > > > > > Guo
> > > > > > > > <
> > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main
> driver
> > > behind
> > > > > > > > > > FLINK-9641
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168
> was
> > > > > > created
> > > > > > > > for
> > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > > integration
> > > > > was
> > > > > > > done
> > > > > > > > > > with
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago
> and
> > > we
> > > > > > ended
> > > > > > > > up
> > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users
> to use
> > > > > Flink
> > > > > > > to
> > > > > > > > > > > process
> > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >
> https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty
> > > simple
> > > > > and
> > > > > > > > there
> > > > > > > > > > is
> > > > > > > > > > > no
> > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have
> > > made a
> > > > > lot
> > > > > > > of
> > > > > > > > > > > changes
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > > first-class
> > > > > > > > citizen in
> > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > >> other computing engines
> for
> > > > > > processing
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how
> to
> > > > > > integrate
> > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> pulsar-flink
> > > > > > > connectors
> > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and
> catalog
> > > API as
> > > > > > the
> > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that
> being
> > > said,
> > > > > > in
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar
> as a
> > > flink
> > > > > > > > catalog
> > > > > > > > > > and
> > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> This is an example about
> how
> > > to
> > > > > use
> > > > > > > > Pulsar
> > > > > > > > > > as
> > > > > > > > > > > a
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a
> blog
> > > post
> > > > > > > > > > explaining
> > > > > > > > > > > > why
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9
> and
> > > what
> > > > > > are
> > > > > > > > the
> > > > > > > > > > > > changes
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not
> just
> > > a
> > > > > > simple
> > > > > > > > data
> > > > > > > > > > > sink
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> > > integrated
> > > > > > > > streaming
> > > > > > > > > > > > data
> > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > > schema/catalog
> > > > > > > and
> > > > > > > > > > > state).
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a
> great
> > > > > > > streaming
> > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified
> data
> > > > > > > > processing.
> > > > > > > > > > > Since
> > > > > > > > > > > > we
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> integration
> > > to
> > > > > > Flink
> > > > > > > > here,
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and
> evolve
> > > the
> > > > > > > > integration
> > > > > > > > > > > > with
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and
> Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit
> more
> > > > > > > background
> > > > > > > > > > about
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know
> what
> > > are
> > > > > > your
> > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at
> 11:54
> > > AM
> > > > > Yun
> > > > > > > > Tang <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar
> becomes
> > > > > more
> > > > > > > and
> > > > > > > > more
> > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people
> willing to
> > > > > > > > contribute to
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > > discussion,
> > > > > > would
> > > > > > > > you
> > > > > > > > > > > please
> > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship between
> this
> > > > > > thread
> > > > > > > to
> > > > > > > > > > > current
> > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > > > connector?
> > > > > > > > Will the
> > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> > > different
> > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September
> 3,
> > > 2019
> > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email]
> <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS]
> Contribute
> > > > > > Pulsar
> > > > > > > > Flink
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > > > > discussion
> > > > > > > of
> > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to
> Flink.
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction
> to
> > > Apache
> > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > > > multi-tenant,
> > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> > > Pulsar
> > > > > > > > includes
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > >>> native support for
> multiple
> > > > > > clusters
> > > > > > > > in a
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication
> of
> > > > > > messages
> > > > > > > > across
> > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> > > seamless
> > > > > > > > > > scalability
> > > > > > > > > > > to
> > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> > > delivery
> > > > > > with
> > > > > > > > > > > persistent
> > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > > BookKeeper.
> > > > > > > > Nowadays,
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar
> Flink
> > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink
> connector
> > > we are
> > > > > > > > planning
> > > > > > > > > > to
> > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar
> > > 2.4.0. The
> > > > > > > main
> > > > > > > > > > > features
> > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> > > source
> > > > > with
> > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results
> to
> > > > > Pulsar
> > > > > > > with
> > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > > exactly-once
> > > > > > as
> > > > > > > > well
> > > > > > > > > > > when
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features
> ready
> > > in its
> > > > > > > 2.5.0
> > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new
> Table
> > > API
> > > > > > Type
> > > > > > > > > > system
> > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> (de)serialize
> > > > > > messages
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink
> new
> > > > > Catalog
> > > > > > > API
> > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> > > tables in
> > > > > > > > Table API
> > > > > > > > > > > as
> > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
>

[Stephan Ewen]

My take would be the following:

  - If we merge the connector now and replace it with a FLIP-27 version
before the 1.10 release, then we need no deprecation process
  - If we don't manage to replace it with a FLIP-27 version before the 1.10
release, than it is good that we have the other version, so no users get
blocked.

In the latter case we can see how we want to do it. Immediate removal of
the old version or deprecation label and keeping it for one more release.
Given that you should be able to use a Flink 1.10 connector with Flink 1.11
as well (stable public APIs) there is also a workaround if you need an old
connector in a newer version. So immediate removal might even be feasible.


On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <[hidden email]> wrote:

> Hi Stephan,
>
> Thanks for the clarification. I completely agree with you and Thomas on the
> process of adding connectors to Flink repo. However, I am wondering what is
> the deprecation process? Given the main concern here was that we may have
> to maintain two Pulsar connector code bases until the old one is removed
> from the repo, it would be good to know how long we have to do that.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]> wrote:
>
> > Some quick thoughts on the connector contribution process. I basically
> > reiterate here what Thomas mentioned in another thread about the Kinesis
> > connector.
> >
> > For connectors, we should favor a low-overhead contribution process, and
> > accept user code and changes more readily than in the core system.
> > That is because connectors have both a big variety of scenarios they get
> > used in (only through use and many small contributions do they become
> > really useful over time) and at the same time, and committers do not use
> > the connector themselves and usually cannot foresee too well what is
> > needed.
> >
> > Further more, a missing connector (or connector feature) is often a
> bigger
> > show stopper for users than a missing API or system feature.
> >
> > Along these lines of thougt, the conclusion would be to take the Pulsar
> > connector now, focus the review on legal/dependencies/rough code style
> and
> > conventions, label it as "beta" (in the sense of "new code" that is "not
> > yet tested through longer use") and go ahead. And then evolve it quickly
> > without putting formal blockers in the way, meaning also adding a new
> FLIP
> > 27 version when it is there.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]> wrote:
> >
> > > Hi Yijie,
> > >
> > > Could you please follow the FLIP process to start a new FLIP
> [DISCUSSION]
> > > thread in the mailing list?
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > >
> > > I see two FLIP-69 discussion in the mailing list now. So there is a
> FLIP
> > > number collision. Can you change the FLIP number to 72?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> wrote:
> > >
> > > > Hi Yijie,
> > > >
> > > > Thanks for sharing the pulsar FLIP.
> > > > Would you mind enabling comments/suggestions on the google doc link?
> > This
> > > > way the contributors from the community can comment on the doc.
> > > >
> > > > Best,
> > > > Rong
> > > >
> > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> [hidden email]>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I've drafted a FLIP that describes the current design of the Pulsar
> > > > > connector:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > >
> > > > > Please take a look and let me know what you think.
> > > > >
> > > > > Thanks,
> > > > > Yijie
> > > > >
> > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Sorry for joining the discussion late and thanks Yijie & Sijie
> for
> > > > > driving
> > > > > > the discussion.
> > > > > > I also think the Pulsar connector would be a very valuable
> addition
> > > to
> > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > >
> > > > > > Regarding the timeline, I also share concerns with Becket on the
> > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > There's also another discussion just started by Stephan on
> dropping
> > > > Kafka
> > > > > > 9/10 support on next Flink release [1].  Although the situation
> is
> > > > > somewhat
> > > > > > different, and Kafka 9/10 connector has been in Flink for almost
> > 3-4
> > > > > years,
> > > > > > based on the discussion I am not sure if a major version release
> > is a
> > > > > > requirement for removing old connector supports.
> > > > > >
> > > > > > I think there shouldn't be a blocker if we agree the old
> connector
> > > will
> > > > > be
> > > > > > removed once FLIP-27 based Pulsar connector is there. As Stephan
> > > > stated,
> > > > > it
> > > > > > is easier to contribute the source sooner and adjust it later.
> > > > > > We should also ensure we clearly communicate the message: for
> > > example,
> > > > > > putting an experimental flag on the pre-FLIP27 connector page of
> > the
> > > > > > website, documentations, etc. Any other thoughts?
> > > > > >
> > > > > > --
> > > > > > Rong
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > > > Technically speaking, removing the old connector code is a
> > > backwards
> > > > > > > incompatible change which requires a major version bump, i.e.
> > Flink
> > > > > 2.x.
> > > > > > > Given that we don't have a clear plan on when to have the next
> > > major
> > > > > > > version release, it seems unclear how long the old connector
> code
> > > > will
> > > > > be
> > > > > > > there if we check it in right now. Or will we remove the old
> > > > connector
> > > > > > > without a major version bump? In any case, it sounds not quite
> > user
> > > > > > > friendly to the those who might use the old Pulsar connector. I
> > am
> > > > not
> > > > > sure
> > > > > > > if it is worth these potential problems in order to have the
> > Pulsar
> > > > > source
> > > > > > > connector checked in one or two months earlier.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <[hidden email]
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Agreed, if we check in the old code, we should make it clear
> > that
> > > > it
> > > > > will
> > > > > > > > be removed as soon as the FLIP-27 based version of the
> > connector
> > > is
> > > > > > > there.
> > > > > > > > We should not commit to maintaining the old version, that
> would
> > > be
> > > > > indeed
> > > > > > > > too much overhead.
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Stephan,
> > > > > > > > >
> > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > >
> > > > > > > > > Yes, the overhead would just be review capacity. In fact, I
> > am
> > > > not
> > > > > > > > worrying
> > > > > > > > > too much about the review capacity. That is just a one time
> > > cost.
> > > > > My
> > > > > > > > > concern is mainly about the long term burden. Assume we
> have
> > > new
> > > > > source
> > > > > > > > > interface ready in 1.10 with newly added Pulsar connectors
> in
> > > old
> > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > interface,
> > > > > the
> > > > > > > old
> > > > > > > > > Pulsar connector might be deprecated almost immediately
> after
> > > > > checked
> > > > > > > in,
> > > > > > > > > but we may still have to maintain two code bases. For the
> > > > existing
> > > > > > > > > connectors, we have to do that anyways. But it would be
> good
> > to
> > > > > avoid
> > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > [hidden email]
> > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all!
> > > > > > > > > >
> > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > connector.
> > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > >
> > > > > > > > > > ## Contribute to Flink or maintain as a community package
> > > > > > > > > >
> > > > > > > > > > Looks like the discussion is more going towards
> > > contribution. I
> > > > > think
> > > > > > > > > that
> > > > > > > > > > is good, especially if we think that we want to build a
> > > > similarly
> > > > > > > deep
> > > > > > > > > > integration with Pulsar as we have for example with
> Kafka.
> > > The
> > > > > > > > connector
> > > > > > > > > > already looks like a more thorough connector than many
> > others
> > > > we
> > > > > have
> > > > > > > > in
> > > > > > > > > > the repository.
> > > > > > > > > >
> > > > > > > > > > With either a repo split, or the new build system, I hope
> > > that
> > > > > the
> > > > > > > > build
> > > > > > > > > > overhead is not a problem.
> > > > > > > > > >
> > > > > > > > > > ## Committer Support
> > > > > > > > > >
> > > > > > > > > > Becket offered some help already, I can also help a bit.
> I
> > > hope
> > > > > that
> > > > > > > > > > between us, we can cover this.
> > > > > > > > > >
> > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > >
> > > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > > progress, but
> > > > > > > > > will
> > > > > > > > > > take 2 more months, I would estimate, before it is fully
> > > > > available.
> > > > > > > > > >
> > > > > > > > > > If we want to be on the safe side with the contribution,
> we
> > > > > should
> > > > > > > > > > contribute the source sooner and adjust it later. That
> > would
> > > > also
> > > > > > > help
> > > > > > > > us
> > > > > > > > > > in case things get crazy towards the 1.10 feature freeze
> > and
> > > it
> > > > > would
> > > > > > > > be
> > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > What would be the overhead of contributing now? Given
> that
> > > the
> > > > > code
> > > > > > > is
> > > > > > > > > > already there, it looks like it would be only review
> > > capacity,
> > > > > right?
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone!
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your attention and the promotion of this
> work.
> > > > > > > > > > >
> > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > specific
> > > > > > > > > discussions.
> > > > > > > > > > >
> > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > consensus.
> > > > > > > > Therefore,
> > > > > > > > > > > I will explain all the functionalities of the existing
> > > > > connector in
> > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > continue
> > > > our
> > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yijie
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > >
> > > > > > > > > > > > If we agree that the goal is to have Pulsar connector
> > in
> > > > > 1.10,
> > > > > > > how
> > > > > > > > > > about
> > > > > > > > > > > we
> > > > > > > > > > > > do the following:
> > > > > > > > > > > >
> > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink main
> > > repo
> > > > > as it
> > > > > > > > is a
> > > > > > > > > > new
> > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > 1. Start to review the Pulsar sink right away as
> there
> > is
> > > > no
> > > > > > > change
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is going
> to
> > > be
> > > > > code
> > > > > > > > > freeze
> > > > > > > > > > in
> > > > > > > > > > > > late Nov and let's say we give a month to the
> > development
> > > > and
> > > > > > > > review
> > > > > > > > > of
> > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late
> Oct.
> > > > There
> > > > > are
> > > > > > > > > still
> > > > > > > > > > 7
> > > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27 is
> > not
> > > > > ready
> > > > > > > by
> > > > > > > > > late
> > > > > > > > > > > > Oct, we can review and check in Pulsar connector with
> > the
> > > > > > > existing
> > > > > > > > > > source
> > > > > > > > > > > > interface. This means we will have Pulsar connector
> in
> > > > Flink
> > > > > > > 1.10,
> > > > > > > > > > either
> > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > >
> > > > > > > > > > > > Because we are going to have Pulsar sink and source
> > > checked
> > > > > in
> > > > > > > > > > > separately,
> > > > > > > > > > > > it might make sense to have two FLIPs, one for Pulsar
> > > sink
> > > > > and
> > > > > > > > > another
> > > > > > > > > > > for
> > > > > > > > > > > > Pulsar source. And we can start the work on Pulsar
> sink
> > > > right
> > > > > > > away.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What's the take from Flink community? Shall we wait
> > for
> > > > > FLIP-27
> > > > > > > > or
> > > > > > > > > > > shall we
> > > > > > > > > > > > > proceed to next steps? And what the next steps are?
> > :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Sijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think having a Pulsar connector in Flink can
> be a
> > > > good
> > > > > > > mutual
> > > > > > > > > > > benefit
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Another perspective is that Pulsar connector is
> the
> > > 1st
> > > > > > > > streaming
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > that integrates with Flink's metadata management
> > > system
> > > > > and
> > > > > > > > > Catalog
> > > > > > > > > > > APIs.
> > > > > > > > > > > > > > It'll be cool to see how the integration turns
> out
> > > and
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > > > need to
> > > > > > > > > > > > > > improve Flink Catalog stack, which are currently
> in
> > > > > Beta, to
> > > > > > > > > cater
> > > > > > > > > > to
> > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> merging
> > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > into
> > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g.
> maybe
> > > one
> > > > > for
> > > > > > > > basic
> > > > > > > > > > > > > > source/sink functionalities and another for
> schema
> > > and
> > > > > > > catalog
> > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27. But
> I
> > > > don't
> > > > > > > think
> > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > should be a blocker in cases where it cannot make
> > its
> > > > way
> > > > > > > into
> > > > > > > > > 1.10
> > > > > > > > > > > or
> > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > committers
> > > > to
> > > > > > > > review
> > > > > > > > > or
> > > > > > > > > > > for
> > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> interfaces.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You are right. It all depends on when the new
> > > source
> > > > > > > > interface
> > > > > > > > > is
> > > > > > > > > > > going
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > be ready. Personally I think it would be there
> in
> > > > > about a
> > > > > > > > month
> > > > > > > > > > or
> > > > > > > > > > > so.
> > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > I could be too optimistic. It would also be
> good
> > to
> > > > > hear
> > > > > > > what
> > > > > > > > > do
> > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > and Stephan think as they are also involved in
> > > > FLIP-27.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In general I think we should have Pulsar
> > connector
> > > in
> > > > > Flink
> > > > > > > > > 1.10,
> > > > > > > > > > > > > > > preferably with the new source interface. We
> can
> > > also
> > > > > check
> > > > > > > > it
> > > > > > > > > in
> > > > > > > > > > > right
> > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > with old source interface, but I suspect few
> > users
> > > > > will use
> > > > > > > > it
> > > > > > > > > > > before
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > reasonable
> > > > > to
> > > > > > > > wait a
> > > > > > > > > > > little
> > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > > interface. As
> > > > > > > > long
> > > > > > > > > > as
> > > > > > > > > > > we
> > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > > doesn't
> > > > > seem
> > > > > > > to
> > > > > > > > > hurt
> > > > > > > > > > > much.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'm wondering what the problem would be if we
> > > > > committed
> > > > > > > the
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > connector before the new source interface is
> > > ready.
> > > > > If I
> > > > > > > > > > > understood
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > correctly, then we need to support the old
> > source
> > > > > > > interface
> > > > > > > > > > > anyway
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > existing connectors. By checking it in early
> I
> > > > could
> > > > > see
> > > > > > > > the
> > > > > > > > > > > benefit
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > our users could start using the connector
> > > earlier.
> > > > > > > > Moreover,
> > > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > > > > > > prevent that the Pulsar integration is being
> > > > delayed
> > > > > in
> > > > > > > > case
> > > > > > > > > > > that the
> > > > > > > > > > > > > > > > source interface should be delayed. The only
> > > > > downside I
> > > > > > > see
> > > > > > > > > is
> > > > > > > > > > > the
> > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > review effort and potential fixes which might
> > be
> > > > > > > irrelevant
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > source interface implementation. I guess it
> > > mainly
> > > > > > > depends
> > > > > > > > on
> > > > > > > > > > how
> > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > we are when the new source interface will be
> > > ready.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > > Although
> > > > > the
> > > > > > > > FLIP
> > > > > > > > > > > wiki
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > > time, a
> > > > > few
> > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > contributors in Flink community were
> actually
> > > > > > > prototyping
> > > > > > > > > the
> > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > We have made some good progress there but
> > want
> > > to
> > > > > > > update
> > > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > after the entire thing is verified to work
> in
> > > > case
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > some
> > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > minute surprise in the implementation. I
> > don't
> > > > > have an
> > > > > > > > > exact
> > > > > > > > > > > ETA
> > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > guess it is going to be within a month or
> so.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I am happy to review the current Flink
> Pulsar
> > > > > connector
> > > > > > > > and
> > > > > > > > > > > see if
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid
> the
> > > > case
> > > > > that
> > > > > > > > we
> > > > > > > > > > > checked
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar connector with some review efforts
> and
> > > > > shortly
> > > > > > > > after
> > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie Shen <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > suggestions!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As Sijie said, the goal of the connector
> > has
> > > > > always
> > > > > > > > been
> > > > > > > > > to
> > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > users with the latest features of both
> > > systems
> > > > as
> > > > > > > soon
> > > > > > > > as
> > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > propose to contribute the connector to
> > Flink
> > > > and
> > > > > hope
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> experts
> > > to
> > > > > ensure
> > > > > > > > the
> > > > > > > > > > > high
> > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence at
> > the
> > > > > > > beginning
> > > > > > > > of
> > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > the connector implementation based on
> Flink
> > > > 1.9;
> > > > > we
> > > > > > > > also
> > > > > > > > > > > wanted
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > build a connector that supports both
> batch
> > > and
> > > > > stream
> > > > > > > > > > > computing
> > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > However, it has been inactive for some
> > time,
> > > so
> > > > > we
> > > > > > > > > decided
> > > > > > > > > > to
> > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > a connector with most of the new
> features,
> > > such
> > > > > as
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > type
> > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > and the new catalog API first. We will
> pay
> > > > > attention
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate it
> > > with
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > as
> > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regarding the test status of the
> connector,
> > > we
> > > > > are
> > > > > > > > > > following
> > > > > > > > > > > the
> > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > connectors' test in Flink repository and
> > > aimed
> > > > to
> > > > > > > > provide
> > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> > hear
> > > > > > > > suggestions
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > > supervision from the Flink community to
> > > improve
> > > > > the
> > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > performance of the connector
> continuously.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie Guo
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > > feedback.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It seems to me that the main question
> > here
> > > is
> > > > > > > about -
> > > > > > > > > > "how
> > > > > > > > > > > can
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > > integration
> > > > > > > is
> > > > > > > > > kind
> > > > > > > > > > > of
> > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > here. I believe there can be many ways
> to
> > > > > achieve
> > > > > > > it.
> > > > > > > > > > > > > > > > > > > As part of the contribution, what we
> are
> > > > > looking
> > > > > > > for
> > > > > > > > > here
> > > > > > > > > > > is
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > communities can build the collaboration
> > > > > > > relationship
> > > > > > > > on
> > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > the integration between Pulsar and
> Flink.
> > > > Even
> > > > > we
> > > > > > > can
> > > > > > > > > try
> > > > > > > > > > > our
> > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > up all the updates in Flink community.
> We
> > > are
> > > > > still
> > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > experiences
> > > > > in
> > > > > > > > Flink
> > > > > > > > > > than
> > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > maintain
> > > > > and
> > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink integration
> > to
> > > > the
> > > > > > > users
> > > > > > > > > who
> > > > > > > > > > > use
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > technologies, we need some help from
> the
> > > > > experts
> > > > > > > from
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27 for a
> > > > while.
> > > > > > > > > Originally
> > > > > > > > > > > we
> > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > of contributing the connectors back
> after
> > > > > > > integrating
> > > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > > decided
> > > > > to
> > > > > > > > > initiate
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > as early as possible. Because we
> believe
> > > > there
> > > > > are
> > > > > > > > more
> > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > > contribution,
> > > > > > > it
> > > > > > > > > can
> > > > > > > > > > > help
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community understand more about Pulsar
> > and
> > > > the
> > > > > > > > > potential
> > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > > verify
> > > > > the
> > > > > > > new
> > > > > > > > > > > connector
> > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> > Qin <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> contributing
> > > the
> > > > > > > Pulsar
> > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > connector
> > > > > with
> > > > > > > > > strong
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> > happy
> > > > the
> > > > > > > > > shepherd
> > > > > > > > > > > this
> > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> provide
> > > > some
> > > > > > > > context
> > > > > > > > > > and
> > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > > connector has
> > > > > > > > hit
> > > > > > > > > > the
> > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > With more and more connectors coming
> > into
> > > > > Flink
> > > > > > > > repo,
> > > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > problems such as long build and
> testing
> > > > > time. To
> > > > > > > > > > address
> > > > > > > > > > > this
> > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > > separate
> > > > > > > > > repository.
> > > > > > > > > > > This
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > on hold due to potential solution to
> > > > shorten
> > > > > the
> > > > > > > > > build
> > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay
> as
> > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > to provide good support for
> > functionality
> > > > and
> > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> Ecosystem
> > > > > project
> > > > > > > > > website
> > > > > > > > > > > and it
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > through some final approval process.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Given the above efforts, it would be
> > > great
> > > > to
> > > > > > > first
> > > > > > > > > see
> > > > > > > > > > > if we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > project
> > > > with
> > > > > > > great
> > > > > > > > > > > support.
> > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > connector
> > > > > is
> > > > > > > > tested
> > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > we can learn something to maintain it
> > as
> > > an
> > > > > > > > ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > quality
> > > > as
> > > > > an
> > > > > > > > > > ecosystem
> > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt it
> > > into
> > > > > the
> > > > > > > main
> > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> FLIP-27
> > > > where
> > > > > we
> > > > > > > are
> > > > > > > > > > > making
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Flink source connector architecture
> and
> > > > > > > interface.
> > > > > > > > > This
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing wise,
> if
> > > we
> > > > > are
> > > > > > > > going
> > > > > > > > > to
> > > > > > > > > > > have
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> wondering
> > if
> > > > we
> > > > > > > should
> > > > > > > > > > hold
> > > > > > > > > > > a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to the
> > new
> > > > > > > interface
> > > > > > > > > to
> > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> Chesnay
> > > > > Schepler <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may end
> up
> > > > > repeating
> > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > > contributing a
> > > > > > > > > > pulsar
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > of which failed because no
> committer
> > > was
> > > > > > > getting
> > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > discussion
> > > > > > > thread
> > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > beforehand and getting several +1's
> > > from
> > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > We should really make sure that if
> we
> > > > > > > > > welcome/approve
> > > > > > > > > > > such
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > contribution it will actually get
> the
> > > > > attention
> > > > > > > > it
> > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to recommend
> > > > > maintaining
> > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from the
> > > > > > > documentation
> > > > > > > > > to
> > > > > > > > > > > give
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > With the upcoming page for sharing
> > > > > artifacts
> > > > > > > > among
> > > > > > > > > > the
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > the state of that anyway?), this
> may
> > > be a
> > > > > > > better
> > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till Rohrmann
> > > wrote:
> > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > > discussion
> > > > > > > > Yijie.
> > > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > connector would be a very
> valuable
> > > > > addition
> > > > > > > > since
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > more popular and it would further
> > > > expand
> > > > > > > > Flink's
> > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > from a project perspective it
> makes
> > > > > sense for
> > > > > > > > me
> > > > > > > > > to
> > > > > > > > > > > place
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > My main concern/question is how
> can
> > > the
> > > > > Flink
> > > > > > > > > > > community
> > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector? We have seen in the
> past
> > > > that
> > > > > > > > > connectors
> > > > > > > > > > > are
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > actively developed components
> > because
> > > > > they
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > > kept
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> > Given
> > > > > that
> > > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> improving
> > > and
> > > > > > > > evolving
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> > this.
> > > > > Hence,
> > > > > > > +1
> > > > > > > > > for
> > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> > Sijie
> > > > Guo
> > > > > <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver
> behind
> > > > > > > FLINK-9641
> > > > > > > > > and
> > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > > created
> > > > > for
> > > > > > > > > bringing
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The integration
> > was
> > > > done
> > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > >> requests about a year ago and we
> > > ended
> > > > > up
> > > > > > > > > > > maintaining
> > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> > Flink
> > > > to
> > > > > > > > process
> > > > > > > > > > > event
> > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > >
> > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty simple
> > and
> > > > > there
> > > > > > > is
> > > > > > > > no
> > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> In the past year, we have made a
> > lot
> > > > of
> > > > > > > > changes
> > > > > > > > > in
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the first-class
> > > > > citizen in
> > > > > > > > > > Pulsar.
> > > > > > > > > > > We
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > processing
> > > > > > > Pulsar
> > > > > > > > > > event
> > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > integrate
> > > > > with
> > > > > > > > Flink
> > > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > connectors
> > > > > from
> > > > > > > > the
> > > > > > > > > > > ground
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog API
> as
> > > the
> > > > > > > > > first-class
> > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >> integration. With that being
> said,
> > > in
> > > > > the
> > > > > > > new
> > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a
> flink
> > > > > catalog
> > > > > > > and
> > > > > > > > > > > query /
> > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> This is an example about how to
> > use
> > > > > Pulsar
> > > > > > > as
> > > > > > > > a
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog
> post
> > > > > > > explaining
> > > > > > > > > why
> > > > > > > > > > we
> > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and
> what
> > > are
> > > > > the
> > > > > > > > > changes
> > > > > > > > > > we
> > > > > > > > > > > > > made
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just a
> > > simple
> > > > > data
> > > > > > > > sink
> > > > > > > > > > or
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> integrated
> > > > > streaming
> > > > > > > > > data
> > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > schema/catalog
> > > > and
> > > > > > > > state).
> > > > > > > > > > The
> > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > > streaming
> > > > > > > > > warehouse
> > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > > processing.
> > > > > > > > Since
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration to
> > > Flink
> > > > > here,
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve the
> > > > > integration
> > > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > > background
> > > > > > > about
> > > > > > > > > the
> > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > >> integration. Let me know what
> are
> > > your
> > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54 AM
> > Yun
> > > > > Tang <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> > more
> > > > and
> > > > > more
> > > > > > > > > > popular
> > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > > contribute to
> > > > > > > > > Flink
> > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Before any further discussion,
> > > would
> > > > > you
> > > > > > > > please
> > > > > > > > > > > give
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > > thread
> > > > to
> > > > > > > > current
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > connector?
> > > > > Will the
> > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally different
> > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > >>>
> ________________________________
> > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3,
> 2019
> > > > 13:57
> > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > > Pulsar
> > > > > Flink
> > > > > > > > > > > connector
> > > > > > > > > > > > > back
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > discussion
> > > > of
> > > > > > > > > > contributing
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to
> Apache
> > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > multi-tenant,
> > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> Pulsar
> > > > > includes
> > > > > > > > > > multiple
> > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > clusters
> > > > > in a
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > messages
> > > > > across
> > > > > > > > > > > clusters,
> > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> seamless
> > > > > > > scalability
> > > > > > > > to
> > > > > > > > > > > over a
> > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message delivery
> > > with
> > > > > > > > persistent
> > > > > > > > > > > message
> > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > >>> provided by Apache BookKeeper.
> > > > > Nowadays,
> > > > > > > > Pulsar
> > > > > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > > connector
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we
> are
> > > > > planning
> > > > > > > to
> > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0.
> The
> > > > main
> > > > > > > > features
> > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming source
> > with
> > > > > > > > > exactly-once
> > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > Pulsar
> > > > with
> > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > >>> would update this to
> exactly-once
> > > as
> > > > > well
> > > > > > > > when
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > >>> transaction features ready in
> its
> > > > 2.5.0
> > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table
> API
> > > Type
> > > > > > > system
> > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > messages
> > > > > with
> > > > > > > the
> > > > > > > > > > help
> > > > > > > > > > > of
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > Catalog
> > > > API
> > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as tables
> in
> > > > > Table API
> > > > > > > > as
> > > > > > > > > > > well as
> > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > >>> [1] https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

[Becket Qin]

Thanks for the explanation, Stephan. I have a few questions / thoughts.

So that means we will remove the old connector without a major version
bump, is that correct?

I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
always work because we saw some dependency class collisions in the past. To
make it safe we may have to maintain the old code for one more release.

To be honest I am still wondering if we have to put the old connector in
Flink repo. if we check in the old connector to Flink. We will end up in
the following situation:
1. Old connector in streamnative/pulsar-flink repo.
2. Old connector in Flink Repo, which may be different from the one in
Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
3. New connector in Flink Repo.

We need to think about how to make the users in each case happy.
- For users of (1), I assume Sijie and Yijie will have to maintain the code
a bit longer for its own compatibility even after we have (2). In that
case, bugs found in old connector may or may not need to be fixed in both
Flink and the streamnative/pulsar-flink repo.
- For users of (2), will we provide bug fixes? If we do, it will be a
little awkward because those bug fixes will be immediately deprecated in
1.11, and removed in 1.12. So we are essentially asking users to migrate
away from the bug fix. After Flink 1.12, users may still have to switch to
use (3) due to the potential dependency class conflicts mentioned above.
- Users of (3) have a much easier life and don't need to worry too much.

The above story seems a little complicated to tell. I think it will be much
easier to not have (2) at all.
1. Old connector in streamnative/pulsar-flink repo.
3. New connector in Flink Repo.

- Old connector will only be maintained in streamnative/pulsar-flink repo
until it is fully deprecated. Users can always use the existing Pulsar
connector in that repo.
- New connector will be in Flink repo and maintained like the other
connectors.

This seems much simpler for users understand and they will not be blocked
from using the old connector. If the concern is about the quality of the
connector in streamnative/pulsar-flink repo, is it enough for us just to
review the code in streamnative/pulsar-flink connector to make sure it
looks good from Flink's perspective?

What do you think?

Thanks,

Jiangjie (Becket) Qin


On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]> wrote:

> My take would be the following:
>
>   - If we merge the connector now and replace it with a FLIP-27 version
> before the 1.10 release, then we need no deprecation process
>   - If we don't manage to replace it with a FLIP-27 version before the 1.10
> release, than it is good that we have the other version, so no users get
> blocked.
>
> In the latter case we can see how we want to do it. Immediate removal of
> the old version or deprecation label and keeping it for one more release.
> Given that you should be able to use a Flink 1.10 connector with Flink 1.11
> as well (stable public APIs) there is also a workaround if you need an old
> connector in a newer version. So immediate removal might even be feasible.
>
>
> On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <[hidden email]> wrote:
>
> > Hi Stephan,
> >
> > Thanks for the clarification. I completely agree with you and Thomas on
> the
> > process of adding connectors to Flink repo. However, I am wondering what
> is
> > the deprecation process? Given the main concern here was that we may have
> > to maintain two Pulsar connector code bases until the old one is removed
> > from the repo, it would be good to know how long we have to do that.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]> wrote:
> >
> > > Some quick thoughts on the connector contribution process. I basically
> > > reiterate here what Thomas mentioned in another thread about the
> Kinesis
> > > connector.
> > >
> > > For connectors, we should favor a low-overhead contribution process,
> and
> > > accept user code and changes more readily than in the core system.
> > > That is because connectors have both a big variety of scenarios they
> get
> > > used in (only through use and many small contributions do they become
> > > really useful over time) and at the same time, and committers do not
> use
> > > the connector themselves and usually cannot foresee too well what is
> > > needed.
> > >
> > > Further more, a missing connector (or connector feature) is often a
> > bigger
> > > show stopper for users than a missing API or system feature.
> > >
> > > Along these lines of thougt, the conclusion would be to take the Pulsar
> > > connector now, focus the review on legal/dependencies/rough code style
> > and
> > > conventions, label it as "beta" (in the sense of "new code" that is
> "not
> > > yet tested through longer use") and go ahead. And then evolve it
> quickly
> > > without putting formal blockers in the way, meaning also adding a new
> > FLIP
> > > 27 version when it is there.
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > >
> > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]>
> wrote:
> > >
> > > > Hi Yijie,
> > > >
> > > > Could you please follow the FLIP process to start a new FLIP
> > [DISCUSSION]
> > > > thread in the mailing list?
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > >
> > > > I see two FLIP-69 discussion in the mailing list now. So there is a
> > FLIP
> > > > number collision. Can you change the FLIP number to 72?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> > wrote:
> > > >
> > > > > Hi Yijie,
> > > > >
> > > > > Thanks for sharing the pulsar FLIP.
> > > > > Would you mind enabling comments/suggestions on the google doc
> link?
> > > This
> > > > > way the contributors from the community can comment on the doc.
> > > > >
> > > > > Best,
> > > > > Rong
> > > > >
> > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I've drafted a FLIP that describes the current design of the
> Pulsar
> > > > > > connector:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > >
> > > > > > Please take a look and let me know what you think.
> > > > > >
> > > > > > Thanks,
> > > > > > Yijie
> > > > > >
> > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <[hidden email]>
> > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Sorry for joining the discussion late and thanks Yijie & Sijie
> > for
> > > > > > driving
> > > > > > > the discussion.
> > > > > > > I also think the Pulsar connector would be a very valuable
> > addition
> > > > to
> > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > >
> > > > > > > Regarding the timeline, I also share concerns with Becket on
> the
> > > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > > There's also another discussion just started by Stephan on
> > dropping
> > > > > Kafka
> > > > > > > 9/10 support on next Flink release [1].  Although the situation
> > is
> > > > > > somewhat
> > > > > > > different, and Kafka 9/10 connector has been in Flink for
> almost
> > > 3-4
> > > > > > years,
> > > > > > > based on the discussion I am not sure if a major version
> release
> > > is a
> > > > > > > requirement for removing old connector supports.
> > > > > > >
> > > > > > > I think there shouldn't be a blocker if we agree the old
> > connector
> > > > will
> > > > > > be
> > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> Stephan
> > > > > stated,
> > > > > > it
> > > > > > > is easier to contribute the source sooner and adjust it later.
> > > > > > > We should also ensure we clearly communicate the message: for
> > > > example,
> > > > > > > putting an experimental flag on the pre-FLIP27 connector page
> of
> > > the
> > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > >
> > > > > > > --
> > > > > > > Rong
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Technically speaking, removing the old connector code is a
> > > > backwards
> > > > > > > > incompatible change which requires a major version bump, i.e.
> > > Flink
> > > > > > 2.x.
> > > > > > > > Given that we don't have a clear plan on when to have the
> next
> > > > major
> > > > > > > > version release, it seems unclear how long the old connector
> > code
> > > > > will
> > > > > > be
> > > > > > > > there if we check it in right now. Or will we remove the old
> > > > > connector
> > > > > > > > without a major version bump? In any case, it sounds not
> quite
> > > user
> > > > > > > > friendly to the those who might use the old Pulsar
> connector. I
> > > am
> > > > > not
> > > > > > sure
> > > > > > > > if it is worth these potential problems in order to have the
> > > Pulsar
> > > > > > source
> > > > > > > > connector checked in one or two months earlier.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Agreed, if we check in the old code, we should make it
> clear
> > > that
> > > > > it
> > > > > > will
> > > > > > > > > be removed as soon as the FLIP-27 based version of the
> > > connector
> > > > is
> > > > > > > > there.
> > > > > > > > > We should not commit to maintaining the old version, that
> > would
> > > > be
> > > > > > indeed
> > > > > > > > > too much overhead.
> > > > > > > > >
> > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Stephan,
> > > > > > > > > >
> > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > >
> > > > > > > > > > Yes, the overhead would just be review capacity. In
> fact, I
> > > am
> > > > > not
> > > > > > > > > worrying
> > > > > > > > > > too much about the review capacity. That is just a one
> time
> > > > cost.
> > > > > > My
> > > > > > > > > > concern is mainly about the long term burden. Assume we
> > have
> > > > new
> > > > > > source
> > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> connectors
> > in
> > > > old
> > > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > > interface,
> > > > > > the
> > > > > > > > old
> > > > > > > > > > Pulsar connector might be deprecated almost immediately
> > after
> > > > > > checked
> > > > > > > > in,
> > > > > > > > > > but we may still have to maintain two code bases. For the
> > > > > existing
> > > > > > > > > > connectors, we have to do that anyways. But it would be
> > good
> > > to
> > > > > > avoid
> > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > [hidden email]
> > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all!
> > > > > > > > > > >
> > > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > > connector.
> > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > >
> > > > > > > > > > > ## Contribute to Flink or maintain as a community
> package
> > > > > > > > > > >
> > > > > > > > > > > Looks like the discussion is more going towards
> > > > contribution. I
> > > > > > think
> > > > > > > > > > that
> > > > > > > > > > > is good, especially if we think that we want to build a
> > > > > similarly
> > > > > > > > deep
> > > > > > > > > > > integration with Pulsar as we have for example with
> > Kafka.
> > > > The
> > > > > > > > > connector
> > > > > > > > > > > already looks like a more thorough connector than many
> > > others
> > > > > we
> > > > > > have
> > > > > > > > > in
> > > > > > > > > > > the repository.
> > > > > > > > > > >
> > > > > > > > > > > With either a repo split, or the new build system, I
> hope
> > > > that
> > > > > > the
> > > > > > > > > build
> > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > >
> > > > > > > > > > > ## Committer Support
> > > > > > > > > > >
> > > > > > > > > > > Becket offered some help already, I can also help a
> bit.
> > I
> > > > hope
> > > > > > that
> > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > >
> > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > >
> > > > > > > > > > > As Becket said, FLIP-27 is actually making some PoC-ing
> > > > > > progress, but
> > > > > > > > > > will
> > > > > > > > > > > take 2 more months, I would estimate, before it is
> fully
> > > > > > available.
> > > > > > > > > > >
> > > > > > > > > > > If we want to be on the safe side with the
> contribution,
> > we
> > > > > > should
> > > > > > > > > > > contribute the source sooner and adjust it later. That
> > > would
> > > > > also
> > > > > > > > help
> > > > > > > > > us
> > > > > > > > > > > in case things get crazy towards the 1.10 feature
> freeze
> > > and
> > > > it
> > > > > > would
> > > > > > > > > be
> > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > What would be the overhead of contributing now? Given
> > that
> > > > the
> > > > > > code
> > > > > > > > is
> > > > > > > > > > > already there, it looks like it would be only review
> > > > capacity,
> > > > > > right?
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Stephan
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your attention and the promotion of this
> > work.
> > > > > > > > > > > >
> > > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > > specific
> > > > > > > > > > discussions.
> > > > > > > > > > > >
> > > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > > consensus.
> > > > > > > > > Therefore,
> > > > > > > > > > > > I will explain all the functionalities of the
> existing
> > > > > > connector in
> > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > > continue
> > > > > our
> > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Yijie
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> connector
> > > in
> > > > > > 1.10,
> > > > > > > > how
> > > > > > > > > > > about
> > > > > > > > > > > > we
> > > > > > > > > > > > > do the following:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink
> main
> > > > repo
> > > > > > as it
> > > > > > > > > is a
> > > > > > > > > > > new
> > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > 1. Start to review the Pulsar sink right away as
> > there
> > > is
> > > > > no
> > > > > > > > change
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is
> going
> > to
> > > > be
> > > > > > code
> > > > > > > > > > freeze
> > > > > > > > > > > in
> > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > development
> > > > > and
> > > > > > > > > review
> > > > > > > > > > of
> > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late
> > Oct.
> > > > > There
> > > > > > are
> > > > > > > > > > still
> > > > > > > > > > > 7
> > > > > > > > > > > > > weeks. Personally I think it is doable. If FLIP-27
> is
> > > not
> > > > > > ready
> > > > > > > > by
> > > > > > > > > > late
> > > > > > > > > > > > > Oct, we can review and check in Pulsar connector
> with
> > > the
> > > > > > > > existing
> > > > > > > > > > > source
> > > > > > > > > > > > > interface. This means we will have Pulsar connector
> > in
> > > > > Flink
> > > > > > > > 1.10,
> > > > > > > > > > > either
> > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Because we are going to have Pulsar sink and source
> > > > checked
> > > > > > in
> > > > > > > > > > > > separately,
> > > > > > > > > > > > > it might make sense to have two FLIPs, one for
> Pulsar
> > > > sink
> > > > > > and
> > > > > > > > > > another
> > > > > > > > > > > > for
> > > > > > > > > > > > > Pulsar source. And we can start the work on Pulsar
> > sink
> > > > > right
> > > > > > > > away.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What's the take from Flink community? Shall we
> wait
> > > for
> > > > > > FLIP-27
> > > > > > > > > or
> > > > > > > > > > > > shall we
> > > > > > > > > > > > > > proceed to next steps? And what the next steps
> are?
> > > :-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think having a Pulsar connector in Flink can
> > be a
> > > > > good
> > > > > > > > mutual
> > > > > > > > > > > > benefit
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Another perspective is that Pulsar connector is
> > the
> > > > 1st
> > > > > > > > > streaming
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > that integrates with Flink's metadata
> management
> > > > system
> > > > > > and
> > > > > > > > > > Catalog
> > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > It'll be cool to see how the integration turns
> > out
> > > > and
> > > > > > > > whether
> > > > > > > > > we
> > > > > > > > > > > > need to
> > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> currently
> > in
> > > > > > Beta, to
> > > > > > > > > > cater
> > > > > > > > > > > to
> > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> > merging
> > > > > > Pulsar
> > > > > > > > > > > connector
> > > > > > > > > > > > into
> > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g.
> > maybe
> > > > one
> > > > > > for
> > > > > > > > > basic
> > > > > > > > > > > > > > > source/sink functionalities and another for
> > schema
> > > > and
> > > > > > > > catalog
> > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27.
> But
> > I
> > > > > don't
> > > > > > > > think
> > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > should be a blocker in cases where it cannot
> make
> > > its
> > > > > way
> > > > > > > > into
> > > > > > > > > > 1.10
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > > committers
> > > > > to
> > > > > > > > > review
> > > > > > > > > > or
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > interfaces.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You are right. It all depends on when the new
> > > > source
> > > > > > > > > interface
> > > > > > > > > > is
> > > > > > > > > > > > going
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > be ready. Personally I think it would be
> there
> > in
> > > > > > about a
> > > > > > > > > month
> > > > > > > > > > > or
> > > > > > > > > > > > so.
> > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > I could be too optimistic. It would also be
> > good
> > > to
> > > > > > hear
> > > > > > > > what
> > > > > > > > > > do
> > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > and Stephan think as they are also involved
> in
> > > > > FLIP-27.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In general I think we should have Pulsar
> > > connector
> > > > in
> > > > > > Flink
> > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > preferably with the new source interface. We
> > can
> > > > also
> > > > > > check
> > > > > > > > > it
> > > > > > > > > > in
> > > > > > > > > > > > right
> > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > with old source interface, but I suspect few
> > > users
> > > > > > will use
> > > > > > > > > it
> > > > > > > > > > > > before
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > > reasonable
> > > > > > to
> > > > > > > > > wait a
> > > > > > > > > > > > little
> > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > to see whether we can jump to the new source
> > > > > > interface. As
> > > > > > > > > long
> > > > > > > > > > > as
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little bit
> > > > doesn't
> > > > > > seem
> > > > > > > > to
> > > > > > > > > > hurt
> > > > > > > > > > > > much.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till Rohrmann
> <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'm wondering what the problem would be if
> we
> > > > > > committed
> > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > connector before the new source interface
> is
> > > > ready.
> > > > > > If I
> > > > > > > > > > > > understood
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > correctly, then we need to support the old
> > > source
> > > > > > > > interface
> > > > > > > > > > > > anyway
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > existing connectors. By checking it in
> early
> > I
> > > > > could
> > > > > > see
> > > > > > > > > the
> > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > our users could start using the connector
> > > > earlier.
> > > > > > > > > Moreover,
> > > > > > > > > > it
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > prevent that the Pulsar integration is
> being
> > > > > delayed
> > > > > > in
> > > > > > > > > case
> > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > source interface should be delayed. The
> only
> > > > > > downside I
> > > > > > > > see
> > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > review effort and potential fixes which
> might
> > > be
> > > > > > > > irrelevant
> > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > source interface implementation. I guess it
> > > > mainly
> > > > > > > > depends
> > > > > > > > > on
> > > > > > > > > > > how
> > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > we are when the new source interface will
> be
> > > > ready.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket Qin <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Just want to have some update on FLIP-27.
> > > > > Although
> > > > > > the
> > > > > > > > > FLIP
> > > > > > > > > > > > wiki
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > discussion thread has been quiet for some
> > > > time, a
> > > > > > few
> > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > contributors in Flink community were
> > actually
> > > > > > > > prototyping
> > > > > > > > > > the
> > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > We have made some good progress there but
> > > want
> > > > to
> > > > > > > > update
> > > > > > > > > > the
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > after the entire thing is verified to
> work
> > in
> > > > > case
> > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > minute surprise in the implementation. I
> > > don't
> > > > > > have an
> > > > > > > > > > exact
> > > > > > > > > > > > ETA
> > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > guess it is going to be within a month or
> > so.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I am happy to review the current Flink
> > Pulsar
> > > > > > connector
> > > > > > > > > and
> > > > > > > > > > > > see if
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to avoid
> > the
> > > > > case
> > > > > > that
> > > > > > > > > we
> > > > > > > > > > > > checked
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar connector with some review efforts
> > and
> > > > > > shortly
> > > > > > > > > after
> > > > > > > > > > > > that
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie
> Shen <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > suggestions!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> connector
> > > has
> > > > > > always
> > > > > > > > > been
> > > > > > > > > > to
> > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > users with the latest features of both
> > > > systems
> > > > > as
> > > > > > > > soon
> > > > > > > > > as
> > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > propose to contribute the connector to
> > > Flink
> > > > > and
> > > > > > hope
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> > experts
> > > > to
> > > > > > ensure
> > > > > > > > > the
> > > > > > > > > > > > high
> > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence
> at
> > > the
> > > > > > > > beginning
> > > > > > > > > of
> > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > the connector implementation based on
> > Flink
> > > > > 1.9;
> > > > > > we
> > > > > > > > > also
> > > > > > > > > > > > wanted
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > build a connector that supports both
> > batch
> > > > and
> > > > > > stream
> > > > > > > > > > > > computing
> > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > However, it has been inactive for some
> > > time,
> > > > so
> > > > > > we
> > > > > > > > > > decided
> > > > > > > > > > > to
> > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > a connector with most of the new
> > features,
> > > > such
> > > > > > as
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > type
> > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > and the new catalog API first. We will
> > pay
> > > > > > attention
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > of FLIP-27 continually and incorporate
> it
> > > > with
> > > > > > the
> > > > > > > > > > > connector
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Regarding the test status of the
> > connector,
> > > > we
> > > > > > are
> > > > > > > > > > > following
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > connectors' test in Flink repository
> and
> > > > aimed
> > > > > to
> > > > > > > > > provide
> > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > tests as we could. We are also happy to
> > > hear
> > > > > > > > > suggestions
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > supervision from the Flink community to
> > > > improve
> > > > > > the
> > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > performance of the connector
> > continuously.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie
> Guo
> > <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments and
> > > > > feedback.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It seems to me that the main question
> > > here
> > > > is
> > > > > > > > about -
> > > > > > > > > > > "how
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 1) I think how and where to host this
> > > > > > integration
> > > > > > > > is
> > > > > > > > > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > here. I believe there can be many
> ways
> > to
> > > > > > achieve
> > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > As part of the contribution, what we
> > are
> > > > > > looking
> > > > > > > > for
> > > > > > > > > > here
> > > > > > > > > > > > is
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > communities can build the
> collaboration
> > > > > > > > relationship
> > > > > > > > > on
> > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > the integration between Pulsar and
> > Flink.
> > > > > Even
> > > > > > we
> > > > > > > > can
> > > > > > > > > > try
> > > > > > > > > > > > our
> > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> community.
> > We
> > > > are
> > > > > > still
> > > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > > experiences
> > > > > > in
> > > > > > > > > Flink
> > > > > > > > > > > than
> > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > > maintain
> > > > > > and
> > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> integration
> > > to
> > > > > the
> > > > > > > > users
> > > > > > > > > > who
> > > > > > > > > > > > use
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > technologies, we need some help from
> > the
> > > > > > experts
> > > > > > > > from
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27
> for a
> > > > > while.
> > > > > > > > > > Originally
> > > > > > > > > > > > we
> > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > of contributing the connectors back
> > after
> > > > > > > > integrating
> > > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But we
> > > > decided
> > > > > > to
> > > > > > > > > > initiate
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > as early as possible. Because we
> > believe
> > > > > there
> > > > > > are
> > > > > > > > > more
> > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > it now rather than later. As part of
> > > > > > contribution,
> > > > > > > > it
> > > > > > > > > > can
> > > > > > > > > > > > help
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community understand more about
> Pulsar
> > > and
> > > > > the
> > > > > > > > > > potential
> > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > Also we can also help Flink community
> > > > verify
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog API).
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM Becket
> > > Qin <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > contributing
> > > > the
> > > > > > > > Pulsar
> > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > > connector
> > > > > > with
> > > > > > > > > > strong
> > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I am
> > > happy
> > > > > the
> > > > > > > > > > shepherd
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> > provide
> > > > > some
> > > > > > > > > context
> > > > > > > > > > > and
> > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > The current way Flink maintains its
> > > > > > connector has
> > > > > > > > > hit
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > With more and more connectors
> coming
> > > into
> > > > > > Flink
> > > > > > > > > repo,
> > > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > problems such as long build and
> > testing
> > > > > > time. To
> > > > > > > > > > > address
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > have attempted to do the following:
> > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into a
> > > > separate
> > > > > > > > > > repository.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > on hold due to potential solution
> to
> > > > > shorten
> > > > > > the
> > > > > > > > > > build
> > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to stay
> > as
> > > > > > ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > functionality
> > > > > and
> > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> > Ecosystem
> > > > > > project
> > > > > > > > > > website
> > > > > > > > > > > > and it
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > through some final approval
> process.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Given the above efforts, it would
> be
> > > > great
> > > > > to
> > > > > > > > first
> > > > > > > > > > see
> > > > > > > > > > > > if we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > > project
> > > > > with
> > > > > > > > great
> > > > > > > > > > > > support.
> > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > > connector
> > > > > > is
> > > > > > > > > tested
> > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > we can learn something to maintain
> it
> > > as
> > > > an
> > > > > > > > > ecosystem
> > > > > > > > > > > > project
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > > quality
> > > > > as
> > > > > > an
> > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well adopt
> it
> > > > into
> > > > > > the
> > > > > > > > main
> > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> > FLIP-27
> > > > > where
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > making
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Flink source connector architecture
> > and
> > > > > > > > interface.
> > > > > > > > > > This
> > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing
> wise,
> > if
> > > > we
> > > > > > are
> > > > > > > > > going
> > > > > > > > > > to
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> > wondering
> > > if
> > > > > we
> > > > > > > > should
> > > > > > > > > > > hold
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to
> the
> > > new
> > > > > > > > interface
> > > > > > > > > > to
> > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> > Chesnay
> > > > > > Schepler <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may end
> > up
> > > > > > repeating
> > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > There were already 2 attempts at
> > > > > > contributing a
> > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > of which failed because no
> > committer
> > > > was
> > > > > > > > getting
> > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > > discussion
> > > > > > > > thread
> > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > beforehand and getting several
> +1's
> > > > from
> > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > We should really make sure that
> if
> > we
> > > > > > > > > > welcome/approve
> > > > > > > > > > > > such
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > contribution it will actually get
> > the
> > > > > > attention
> > > > > > > > > it
> > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> recommend
> > > > > > maintaining
> > > > > > > > > the
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from
> the
> > > > > > > > documentation
> > > > > > > > > > to
> > > > > > > > > > > > give
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> sharing
> > > > > > artifacts
> > > > > > > > > among
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > the state of that anyway?), this
> > may
> > > > be a
> > > > > > > > better
> > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> Rohrmann
> > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting this
> > > > > discussion
> > > > > > > > > Yijie.
> > > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> > valuable
> > > > > > addition
> > > > > > > > > since
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> further
> > > > > expand
> > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > from a project perspective it
> > makes
> > > > > > sense for
> > > > > > > > > me
> > > > > > > > > > to
> > > > > > > > > > > > place
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > My main concern/question is how
> > can
> > > > the
> > > > > > Flink
> > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in the
> > past
> > > > > that
> > > > > > > > > > connectors
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > actively developed components
> > > because
> > > > > > they
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > external system and with Flink.
> > > Given
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> > improving
> > > > and
> > > > > > > > > evolving
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > optimistic that we can achieve
> > > this.
> > > > > > Hence,
> > > > > > > > +1
> > > > > > > > > > for
> > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03 AM
> > > Sijie
> > > > > Guo
> > > > > > <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver
> > behind
> > > > > > > > FLINK-9641
> > > > > > > > > > and
> > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168 was
> > > > created
> > > > > > for
> > > > > > > > > > bringing
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> integration
> > > was
> > > > > done
> > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago and
> we
> > > > ended
> > > > > > up
> > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to use
> > > Flink
> > > > > to
> > > > > > > > > process
> > > > > > > > > > > > event
> > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > >
> > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty
> simple
> > > and
> > > > > > there
> > > > > > > > is
> > > > > > > > > no
> > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have
> made a
> > > lot
> > > > > of
> > > > > > > > > changes
> > > > > > > > > > in
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> first-class
> > > > > > citizen in
> > > > > > > > > > > Pulsar.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > > processing
> > > > > > > > Pulsar
> > > > > > > > > > > event
> > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > > integrate
> > > > > > with
> > > > > > > > > Flink
> > > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > > connectors
> > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > ground
> > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog
> API
> > as
> > > > the
> > > > > > > > > > first-class
> > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > >> integration. With that being
> > said,
> > > > in
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a
> > flink
> > > > > > catalog
> > > > > > > > and
> > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> This is an example about how
> to
> > > use
> > > > > > Pulsar
> > > > > > > > as
> > > > > > > > > a
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a blog
> > post
> > > > > > > > explaining
> > > > > > > > > > why
> > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and
> > what
> > > > are
> > > > > > the
> > > > > > > > > > changes
> > > > > > > > > > > we
> > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not just
> a
> > > > simple
> > > > > > data
> > > > > > > > > sink
> > > > > > > > > > > or
> > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> > integrated
> > > > > > streaming
> > > > > > > > > > data
> > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > schema/catalog
> > > > > and
> > > > > > > > > state).
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a great
> > > > > streaming
> > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified data
> > > > > > processing.
> > > > > > > > > Since
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar integration
> to
> > > > Flink
> > > > > > here,
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve
> the
> > > > > > integration
> > > > > > > > > > with
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and Pulsar.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit more
> > > > > background
> > > > > > > > about
> > > > > > > > > > the
> > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know what
> > are
> > > > your
> > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54
> AM
> > > Yun
> > > > > > Tang <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar becomes
> > > more
> > > > > and
> > > > > > more
> > > > > > > > > > > popular
> > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > >>> to see more people willing to
> > > > > > contribute to
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> discussion,
> > > > would
> > > > > > you
> > > > > > > > > please
> > > > > > > > > > > > give
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > >>> the relationship between this
> > > > thread
> > > > > to
> > > > > > > > > current
> > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > connector?
> > > > > > Will the
> > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> different
> > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > >>>
> > ________________________________
> > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3,
> > 2019
> > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS] Contribute
> > > > Pulsar
> > > > > > Flink
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > > discussion
> > > > > of
> > > > > > > > > > > contributing
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to Flink.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to
> > Apache
> > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > multi-tenant,
> > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> > Pulsar
> > > > > > includes
> > > > > > > > > > > multiple
> > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > > clusters
> > > > > > in a
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > > messages
> > > > > > across
> > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> > seamless
> > > > > > > > scalability
> > > > > > > > > to
> > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> delivery
> > > > with
> > > > > > > > > persistent
> > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> BookKeeper.
> > > > > > Nowadays,
> > > > > > > > > Pulsar
> > > > > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar Flink
> > > > > connector
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector we
> > are
> > > > > > planning
> > > > > > > > to
> > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar 2.4.0.
> > The
> > > > > main
> > > > > > > > > features
> > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> source
> > > with
> > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > > Pulsar
> > > > > with
> > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > exactly-once
> > > > as
> > > > > > well
> > > > > > > > > when
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > >>> transaction features ready in
> > its
> > > > > 2.5.0
> > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new Table
> > API
> > > > Type
> > > > > > > > system
> > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > > messages
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > help
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > > Catalog
> > > > > API
> > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> tables
> > in
> > > > > > Table API
> > > > > > > > > as
> > > > > > > > > > > > well as
> > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > >>> [1]
> https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Sijie Guo-2]

Thanks everyone here. Sorry for jumping into the discussion here.

I am not very familiar about the deprecation process in Flink. If I
misunderstood the process, please fix me.

As far as I understand, FLIP-27 is introducing a new unified API for
connectors. After it introduces the new API
and before moving all the existing connectors from old API to new API, both
old ApI and new API will co-exist
for a while until Flink moves all existing connectors to new API. So the
Pulsar connector (using old API) can
follow the deprecation process with other connector using old API and the
deprecation of old API, no?

If that's the case, I think contributing the current connector back to
Flink rather than maintaining it outside Flink
would provide a bit more benefits. We can deprecate the existing
streamnative/pulsar-flink repo and point the users
to use the connector in Flink repo. So all the review processes will happen
within Flink for both old connector and
new connector. It also reduces the confusions for the users as the
documentation and code base happen in one place.

Thoughts?

- Sijie




On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <[hidden email]> wrote:

> Thanks for the explanation, Stephan. I have a few questions / thoughts.
>
> So that means we will remove the old connector without a major version
> bump, is that correct?
>
> I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
> always work because we saw some dependency class collisions in the past. To
> make it safe we may have to maintain the old code for one more release.
>
> To be honest I am still wondering if we have to put the old connector in
> Flink repo. if we check in the old connector to Flink. We will end up in
> the following situation:
> 1. Old connector in streamnative/pulsar-flink repo.
> 2. Old connector in Flink Repo, which may be different from the one in
> Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> 3. New connector in Flink Repo.
>
> We need to think about how to make the users in each case happy.
> - For users of (1), I assume Sijie and Yijie will have to maintain the code
> a bit longer for its own compatibility even after we have (2). In that
> case, bugs found in old connector may or may not need to be fixed in both
> Flink and the streamnative/pulsar-flink repo.
> - For users of (2), will we provide bug fixes? If we do, it will be a
> little awkward because those bug fixes will be immediately deprecated in
> 1.11, and removed in 1.12. So we are essentially asking users to migrate
> away from the bug fix. After Flink 1.12, users may still have to switch to
> use (3) due to the potential dependency class conflicts mentioned above.
> - Users of (3) have a much easier life and don't need to worry too much.
>
> The above story seems a little complicated to tell. I think it will be much
> easier to not have (2) at all.
> 1. Old connector in streamnative/pulsar-flink repo.
> 3. New connector in Flink Repo.
>
> - Old connector will only be maintained in streamnative/pulsar-flink repo
> until it is fully deprecated. Users can always use the existing Pulsar
> connector in that repo.
> - New connector will be in Flink repo and maintained like the other
> connectors.
>
> This seems much simpler for users understand and they will not be blocked
> from using the old connector. If the concern is about the quality of the
> connector in streamnative/pulsar-flink repo, is it enough for us just to
> review the code in streamnative/pulsar-flink connector to make sure it
> looks good from Flink's perspective?
>
> What do you think?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]> wrote:
>
> > My take would be the following:
> >
> >   - If we merge the connector now and replace it with a FLIP-27 version
> > before the 1.10 release, then we need no deprecation process
> >   - If we don't manage to replace it with a FLIP-27 version before the
> 1.10
> > release, than it is good that we have the other version, so no users get
> > blocked.
> >
> > In the latter case we can see how we want to do it. Immediate removal of
> > the old version or deprecation label and keeping it for one more release.
> > Given that you should be able to use a Flink 1.10 connector with Flink
> 1.11
> > as well (stable public APIs) there is also a workaround if you need an
> old
> > connector in a newer version. So immediate removal might even be
> feasible.
> >
> >
> > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <[hidden email]>
> wrote:
> >
> > > Hi Stephan,
> > >
> > > Thanks for the clarification. I completely agree with you and Thomas on
> > the
> > > process of adding connectors to Flink repo. However, I am wondering
> what
> > is
> > > the deprecation process? Given the main concern here was that we may
> have
> > > to maintain two Pulsar connector code bases until the old one is
> removed
> > > from the repo, it would be good to know how long we have to do that.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]> wrote:
> > >
> > > > Some quick thoughts on the connector contribution process. I
> basically
> > > > reiterate here what Thomas mentioned in another thread about the
> > Kinesis
> > > > connector.
> > > >
> > > > For connectors, we should favor a low-overhead contribution process,
> > and
> > > > accept user code and changes more readily than in the core system.
> > > > That is because connectors have both a big variety of scenarios they
> > get
> > > > used in (only through use and many small contributions do they become
> > > > really useful over time) and at the same time, and committers do not
> > use
> > > > the connector themselves and usually cannot foresee too well what is
> > > > needed.
> > > >
> > > > Further more, a missing connector (or connector feature) is often a
> > > bigger
> > > > show stopper for users than a missing API or system feature.
> > > >
> > > > Along these lines of thougt, the conclusion would be to take the
> Pulsar
> > > > connector now, focus the review on legal/dependencies/rough code
> style
> > > and
> > > > conventions, label it as "beta" (in the sense of "new code" that is
> > "not
> > > > yet tested through longer use") and go ahead. And then evolve it
> > quickly
> > > > without putting formal blockers in the way, meaning also adding a new
> > > FLIP
> > > > 27 version when it is there.
> > > >
> > > > Best,
> > > > Stephan
> > > >
> > > >
> > > >
> > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]>
> > wrote:
> > > >
> > > > > Hi Yijie,
> > > > >
> > > > > Could you please follow the FLIP process to start a new FLIP
> > > [DISCUSSION]
> > > > > thread in the mailing list?
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > >
> > > > > I see two FLIP-69 discussion in the mailing list now. So there is a
> > > FLIP
> > > > > number collision. Can you change the FLIP number to 72?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hi Yijie,
> > > > > >
> > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > Would you mind enabling comments/suggestions on the google doc
> > link?
> > > > This
> > > > > > way the contributors from the community can comment on the doc.
> > > > > >
> > > > > > Best,
> > > > > > Rong
> > > > > >
> > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > [hidden email]>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > I've drafted a FLIP that describes the current design of the
> > Pulsar
> > > > > > > connector:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > >
> > > > > > > Please take a look and let me know what you think.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yijie
> > > > > > >
> > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> [hidden email]>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Sorry for joining the discussion late and thanks Yijie &
> Sijie
> > > for
> > > > > > > driving
> > > > > > > > the discussion.
> > > > > > > > I also think the Pulsar connector would be a very valuable
> > > addition
> > > > > to
> > > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > > >
> > > > > > > > Regarding the timeline, I also share concerns with Becket on
> > the
> > > > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > > > There's also another discussion just started by Stephan on
> > > dropping
> > > > > > Kafka
> > > > > > > > 9/10 support on next Flink release [1].  Although the
> situation
> > > is
> > > > > > > somewhat
> > > > > > > > different, and Kafka 9/10 connector has been in Flink for
> > almost
> > > > 3-4
> > > > > > > years,
> > > > > > > > based on the discussion I am not sure if a major version
> > release
> > > > is a
> > > > > > > > requirement for removing old connector supports.
> > > > > > > >
> > > > > > > > I think there shouldn't be a blocker if we agree the old
> > > connector
> > > > > will
> > > > > > > be
> > > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> > Stephan
> > > > > > stated,
> > > > > > > it
> > > > > > > > is easier to contribute the source sooner and adjust it
> later.
> > > > > > > > We should also ensure we clearly communicate the message: for
> > > > > example,
> > > > > > > > putting an experimental flag on the pre-FLIP27 connector page
> > of
> > > > the
> > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rong
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Technically speaking, removing the old connector code is a
> > > > > backwards
> > > > > > > > > incompatible change which requires a major version bump,
> i.e.
> > > > Flink
> > > > > > > 2.x.
> > > > > > > > > Given that we don't have a clear plan on when to have the
> > next
> > > > > major
> > > > > > > > > version release, it seems unclear how long the old
> connector
> > > code
> > > > > > will
> > > > > > > be
> > > > > > > > > there if we check it in right now. Or will we remove the
> old
> > > > > > connector
> > > > > > > > > without a major version bump? In any case, it sounds not
> > quite
> > > > user
> > > > > > > > > friendly to the those who might use the old Pulsar
> > connector. I
> > > > am
> > > > > > not
> > > > > > > sure
> > > > > > > > > if it is worth these potential problems in order to have
> the
> > > > Pulsar
> > > > > > > source
> > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Agreed, if we check in the old code, we should make it
> > clear
> > > > that
> > > > > > it
> > > > > > > will
> > > > > > > > > > be removed as soon as the FLIP-27 based version of the
> > > > connector
> > > > > is
> > > > > > > > > there.
> > > > > > > > > > We should not commit to maintaining the old version, that
> > > would
> > > > > be
> > > > > > > indeed
> > > > > > > > > > too much overhead.
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Stephan,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > >
> > > > > > > > > > > Yes, the overhead would just be review capacity. In
> > fact, I
> > > > am
> > > > > > not
> > > > > > > > > > worrying
> > > > > > > > > > > too much about the review capacity. That is just a one
> > time
> > > > > cost.
> > > > > > > My
> > > > > > > > > > > concern is mainly about the long term burden. Assume we
> > > have
> > > > > new
> > > > > > > source
> > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > connectors
> > > in
> > > > > old
> > > > > > > > > > > interface. Later on if we migrate Pulsar to new source
> > > > > interface,
> > > > > > > the
> > > > > > > > > old
> > > > > > > > > > > Pulsar connector might be deprecated almost immediately
> > > after
> > > > > > > checked
> > > > > > > > > in,
> > > > > > > > > > > but we may still have to maintain two code bases. For
> the
> > > > > > existing
> > > > > > > > > > > connectors, we have to do that anyways. But it would be
> > > good
> > > > to
> > > > > > > avoid
> > > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > [hidden email]
> > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all!
> > > > > > > > > > > >
> > > > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > > > connector.
> > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > >
> > > > > > > > > > > > ## Contribute to Flink or maintain as a community
> > package
> > > > > > > > > > > >
> > > > > > > > > > > > Looks like the discussion is more going towards
> > > > > contribution. I
> > > > > > > think
> > > > > > > > > > > that
> > > > > > > > > > > > is good, especially if we think that we want to
> build a
> > > > > > similarly
> > > > > > > > > deep
> > > > > > > > > > > > integration with Pulsar as we have for example with
> > > Kafka.
> > > > > The
> > > > > > > > > > connector
> > > > > > > > > > > > already looks like a more thorough connector than
> many
> > > > others
> > > > > > we
> > > > > > > have
> > > > > > > > > > in
> > > > > > > > > > > > the repository.
> > > > > > > > > > > >
> > > > > > > > > > > > With either a repo split, or the new build system, I
> > hope
> > > > > that
> > > > > > > the
> > > > > > > > > > build
> > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > >
> > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > >
> > > > > > > > > > > > Becket offered some help already, I can also help a
> > bit.
> > > I
> > > > > hope
> > > > > > > that
> > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > >
> > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > >
> > > > > > > > > > > > As Becket said, FLIP-27 is actually making some
> PoC-ing
> > > > > > > progress, but
> > > > > > > > > > > will
> > > > > > > > > > > > take 2 more months, I would estimate, before it is
> > fully
> > > > > > > available.
> > > > > > > > > > > >
> > > > > > > > > > > > If we want to be on the safe side with the
> > contribution,
> > > we
> > > > > > > should
> > > > > > > > > > > > contribute the source sooner and adjust it later.
> That
> > > > would
> > > > > > also
> > > > > > > > > help
> > > > > > > > > > us
> > > > > > > > > > > > in case things get crazy towards the 1.10 feature
> > freeze
> > > > and
> > > > > it
> > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > What would be the overhead of contributing now? Given
> > > that
> > > > > the
> > > > > > > code
> > > > > > > > > is
> > > > > > > > > > > > already there, it looks like it would be only review
> > > > > capacity,
> > > > > > > right?
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Stephan
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your attention and the promotion of this
> > > work.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We will prepare a FLIP as soon as possible for more
> > > > > specific
> > > > > > > > > > > discussions.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > > > consensus.
> > > > > > > > > > Therefore,
> > > > > > > > > > > > > I will explain all the functionalities of the
> > existing
> > > > > > > connector in
> > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > > > continue
> > > > > > our
> > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Yijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> > connector
> > > > in
> > > > > > > 1.10,
> > > > > > > > > how
> > > > > > > > > > > > about
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to Flink
> > main
> > > > > repo
> > > > > > > as it
> > > > > > > > > > is a
> > > > > > > > > > > > new
> > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away as
> > > there
> > > > is
> > > > > > no
> > > > > > > > > change
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is
> > going
> > > to
> > > > > be
> > > > > > > code
> > > > > > > > > > > freeze
> > > > > > > > > > > > in
> > > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > > development
> > > > > > and
> > > > > > > > > > review
> > > > > > > > > > > of
> > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by late
> > > Oct.
> > > > > > There
> > > > > > > are
> > > > > > > > > > > still
> > > > > > > > > > > > 7
> > > > > > > > > > > > > > weeks. Personally I think it is doable. If
> FLIP-27
> > is
> > > > not
> > > > > > > ready
> > > > > > > > > by
> > > > > > > > > > > late
> > > > > > > > > > > > > > Oct, we can review and check in Pulsar connector
> > with
> > > > the
> > > > > > > > > existing
> > > > > > > > > > > > source
> > > > > > > > > > > > > > interface. This means we will have Pulsar
> connector
> > > in
> > > > > > Flink
> > > > > > > > > 1.10,
> > > > > > > > > > > > either
> > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Because we are going to have Pulsar sink and
> source
> > > > > checked
> > > > > > > in
> > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > it might make sense to have two FLIPs, one for
> > Pulsar
> > > > > sink
> > > > > > > and
> > > > > > > > > > > another
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > Pulsar source. And we can start the work on
> Pulsar
> > > sink
> > > > > > right
> > > > > > > > > away.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What's the take from Flink community? Shall we
> > wait
> > > > for
> > > > > > > FLIP-27
> > > > > > > > > > or
> > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > proceed to next steps? And what the next steps
> > are?
> > > > :-)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think having a Pulsar connector in Flink
> can
> > > be a
> > > > > > good
> > > > > > > > > mutual
> > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Another perspective is that Pulsar connector
> is
> > > the
> > > > > 1st
> > > > > > > > > > streaming
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > management
> > > > > system
> > > > > > > and
> > > > > > > > > > > Catalog
> > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > It'll be cool to see how the integration
> turns
> > > out
> > > > > and
> > > > > > > > > whether
> > > > > > > > > > we
> > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> > currently
> > > in
> > > > > > > Beta, to
> > > > > > > > > > > cater
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> > > merging
> > > > > > > Pulsar
> > > > > > > > > > > > connector
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs, e.g.
> > > maybe
> > > > > one
> > > > > > > for
> > > > > > > > > > basic
> > > > > > > > > > > > > > > > source/sink functionalities and another for
> > > schema
> > > > > and
> > > > > > > > > catalog
> > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for FLIP-27.
> > But
> > > I
> > > > > > don't
> > > > > > > > > think
> > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > should be a blocker in cases where it cannot
> > make
> > > > its
> > > > > > way
> > > > > > > > > into
> > > > > > > > > > > 1.10
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > > > committers
> > > > > > to
> > > > > > > > > > review
> > > > > > > > > > > or
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > > interfaces.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > You are right. It all depends on when the
> new
> > > > > source
> > > > > > > > > > interface
> > > > > > > > > > > is
> > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > be ready. Personally I think it would be
> > there
> > > in
> > > > > > > about a
> > > > > > > > > > month
> > > > > > > > > > > > or
> > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > I could be too optimistic. It would also be
> > > good
> > > > to
> > > > > > > hear
> > > > > > > > > what
> > > > > > > > > > > do
> > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > and Stephan think as they are also involved
> > in
> > > > > > FLIP-27.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > In general I think we should have Pulsar
> > > > connector
> > > > > in
> > > > > > > Flink
> > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > preferably with the new source interface.
> We
> > > can
> > > > > also
> > > > > > > check
> > > > > > > > > > it
> > > > > > > > > > > in
> > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > with old source interface, but I suspect
> few
> > > > users
> > > > > > > will use
> > > > > > > > > > it
> > > > > > > > > > > > > before
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > next official release. Therefore, it seems
> > > > > reasonable
> > > > > > > to
> > > > > > > > > > wait a
> > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > to see whether we can jump to the new
> source
> > > > > > > interface. As
> > > > > > > > > > long
> > > > > > > > > > > > as
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little
> bit
> > > > > doesn't
> > > > > > > seem
> > > > > > > > > to
> > > > > > > > > > > hurt
> > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till
> Rohrmann
> > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'm wondering what the problem would be
> if
> > we
> > > > > > > committed
> > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > connector before the new source interface
> > is
> > > > > ready.
> > > > > > > If I
> > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > correctly, then we need to support the
> old
> > > > source
> > > > > > > > > interface
> > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > existing connectors. By checking it in
> > early
> > > I
> > > > > > could
> > > > > > > see
> > > > > > > > > > the
> > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > our users could start using the connector
> > > > > earlier.
> > > > > > > > > > Moreover,
> > > > > > > > > > > it
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > prevent that the Pulsar integration is
> > being
> > > > > > delayed
> > > > > > > in
> > > > > > > > > > case
> > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > source interface should be delayed. The
> > only
> > > > > > > downside I
> > > > > > > > > see
> > > > > > > > > > > is
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > review effort and potential fixes which
> > might
> > > > be
> > > > > > > > > irrelevant
> > > > > > > > > > > for
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > source interface implementation. I guess
> it
> > > > > mainly
> > > > > > > > > depends
> > > > > > > > > > on
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > we are when the new source interface will
> > be
> > > > > ready.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket
> Qin <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Just want to have some update on
> FLIP-27.
> > > > > > Although
> > > > > > > the
> > > > > > > > > > FLIP
> > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > discussion thread has been quiet for
> some
> > > > > time, a
> > > > > > > few
> > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > contributors in Flink community were
> > > actually
> > > > > > > > > prototyping
> > > > > > > > > > > the
> > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > We have made some good progress there
> but
> > > > want
> > > > > to
> > > > > > > > > update
> > > > > > > > > > > the
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > after the entire thing is verified to
> > work
> > > in
> > > > > > case
> > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > minute surprise in the implementation.
> I
> > > > don't
> > > > > > > have an
> > > > > > > > > > > exact
> > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > guess it is going to be within a month
> or
> > > so.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I am happy to review the current Flink
> > > Pulsar
> > > > > > > connector
> > > > > > > > > > and
> > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to
> avoid
> > > the
> > > > > > case
> > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar connector with some review
> efforts
> > > and
> > > > > > > shortly
> > > > > > > > > > after
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie
> > Shen <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > > suggestions!
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> > connector
> > > > has
> > > > > > > always
> > > > > > > > > > been
> > > > > > > > > > > to
> > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > users with the latest features of
> both
> > > > > systems
> > > > > > as
> > > > > > > > > soon
> > > > > > > > > > as
> > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > propose to contribute the connector
> to
> > > > Flink
> > > > > > and
> > > > > > > hope
> > > > > > > > > > to
> > > > > > > > > > > > get
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> > > experts
> > > > > to
> > > > > > > ensure
> > > > > > > > > > the
> > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its existence
> > at
> > > > the
> > > > > > > > > beginning
> > > > > > > > > > of
> > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > the connector implementation based on
> > > Flink
> > > > > > 1.9;
> > > > > > > we
> > > > > > > > > > also
> > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > build a connector that supports both
> > > batch
> > > > > and
> > > > > > > stream
> > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > However, it has been inactive for
> some
> > > > time,
> > > > > so
> > > > > > > we
> > > > > > > > > > > decided
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > a connector with most of the new
> > > features,
> > > > > such
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > and the new catalog API first. We
> will
> > > pay
> > > > > > > attention
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> incorporate
> > it
> > > > > with
> > > > > > > the
> > > > > > > > > > > > connector
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Regarding the test status of the
> > > connector,
> > > > > we
> > > > > > > are
> > > > > > > > > > > > following
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > connectors' test in Flink repository
> > and
> > > > > aimed
> > > > > > to
> > > > > > > > > > provide
> > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > tests as we could. We are also happy
> to
> > > > hear
> > > > > > > > > > suggestions
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > supervision from the Flink community
> to
> > > > > improve
> > > > > > > the
> > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > performance of the connector
> > > continuously.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM Sijie
> > Guo
> > > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments
> and
> > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It seems to me that the main
> question
> > > > here
> > > > > is
> > > > > > > > > about -
> > > > > > > > > > > > "how
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community maintain the connector?".
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Here are two thoughts from myself.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 1) I think how and where to host
> this
> > > > > > > integration
> > > > > > > > > is
> > > > > > > > > > > kind
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > here. I believe there can be many
> > ways
> > > to
> > > > > > > achieve
> > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > As part of the contribution, what
> we
> > > are
> > > > > > > looking
> > > > > > > > > for
> > > > > > > > > > > here
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > communities can build the
> > collaboration
> > > > > > > > > relationship
> > > > > > > > > > on
> > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > the integration between Pulsar and
> > > Flink.
> > > > > > Even
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > try
> > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > community.
> > > We
> > > > > are
> > > > > > > still
> > > > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > > > experiences
> > > > > > > in
> > > > > > > > > > Flink
> > > > > > > > > > > > than
> > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community. In order to make sure we
> > > > > maintain
> > > > > > > and
> > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > integration
> > > > to
> > > > > > the
> > > > > > > > > users
> > > > > > > > > > > who
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > technologies, we need some help
> from
> > > the
> > > > > > > experts
> > > > > > > > > from
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27
> > for a
> > > > > > while.
> > > > > > > > > > > Originally
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > of contributing the connectors back
> > > after
> > > > > > > > > integrating
> > > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27. But
> we
> > > > > decided
> > > > > > > to
> > > > > > > > > > > initiate
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > as early as possible. Because we
> > > believe
> > > > > > there
> > > > > > > are
> > > > > > > > > > more
> > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > it now rather than later. As part
> of
> > > > > > > contribution,
> > > > > > > > > it
> > > > > > > > > > > can
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community understand more about
> > Pulsar
> > > > and
> > > > > > the
> > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > Also we can also help Flink
> community
> > > > > verify
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog
> API).
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM
> Becket
> > > > Qin <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > > contributing
> > > > > the
> > > > > > > > > Pulsar
> > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > In general, I think having Pulsar
> > > > > connector
> > > > > > > with
> > > > > > > > > > > strong
> > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So I
> am
> > > > happy
> > > > > > the
> > > > > > > > > > > shepherd
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> > > provide
> > > > > > some
> > > > > > > > > > context
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The current way Flink maintains
> its
> > > > > > > connector has
> > > > > > > > > > hit
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > With more and more connectors
> > coming
> > > > into
> > > > > > > Flink
> > > > > > > > > > repo,
> > > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > problems such as long build and
> > > testing
> > > > > > > time. To
> > > > > > > > > > > > address
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > have attempted to do the
> following:
> > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors into
> a
> > > > > separate
> > > > > > > > > > > repository.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > on hold due to potential solution
> > to
> > > > > > shorten
> > > > > > > the
> > > > > > > > > > > build
> > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to
> stay
> > > as
> > > > > > > ecosystem
> > > > > > > > > > > > project
> > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > > functionality
> > > > > > and
> > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> > > Ecosystem
> > > > > > > project
> > > > > > > > > > > website
> > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > through some final approval
> > process.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it would
> > be
> > > > > great
> > > > > > to
> > > > > > > > > first
> > > > > > > > > > > see
> > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an ecosystem
> > > > project
> > > > > > with
> > > > > > > > > great
> > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink Pulsar
> > > > > connector
> > > > > > > is
> > > > > > > > > > tested
> > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > we can learn something to
> maintain
> > it
> > > > as
> > > > > an
> > > > > > > > > > ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If the
> > > > quality
> > > > > > as
> > > > > > > an
> > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well
> adopt
> > it
> > > > > into
> > > > > > > the
> > > > > > > > > main
> > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> > > FLIP-27
> > > > > > where
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Flink source connector
> architecture
> > > and
> > > > > > > > > interface.
> > > > > > > > > > > This
> > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing
> > wise,
> > > if
> > > > > we
> > > > > > > are
> > > > > > > > > > going
> > > > > > > > > > > to
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> > > wondering
> > > > if
> > > > > > we
> > > > > > > > > should
> > > > > > > > > > > > hold
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt to
> > the
> > > > new
> > > > > > > > > interface
> > > > > > > > > > > to
> > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> > > Chesnay
> > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may
> end
> > > up
> > > > > > > repeating
> > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > There were already 2 attempts
> at
> > > > > > > contributing a
> > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > of which failed because no
> > > committer
> > > > > was
> > > > > > > > > getting
> > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > contributor opening a dedicated
> > > > > > discussion
> > > > > > > > > thread
> > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > beforehand and getting several
> > +1's
> > > > > from
> > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > We should really make sure that
> > if
> > > we
> > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > contribution it will actually
> get
> > > the
> > > > > > > attention
> > > > > > > > > > it
> > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> > recommend
> > > > > > > maintaining
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it from
> > the
> > > > > > > > > documentation
> > > > > > > > > > > to
> > > > > > > > > > > > > give
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> > sharing
> > > > > > > artifacts
> > > > > > > > > > among
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?),
> this
> > > may
> > > > > be a
> > > > > > > > > better
> > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> > Rohrmann
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting
> this
> > > > > > discussion
> > > > > > > > > > Yijie.
> > > > > > > > > > > I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> > > valuable
> > > > > > > addition
> > > > > > > > > > since
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> > further
> > > > > > expand
> > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > from a project perspective it
> > > makes
> > > > > > > sense for
> > > > > > > > > > me
> > > > > > > > > > > to
> > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > My main concern/question is
> how
> > > can
> > > > > the
> > > > > > > Flink
> > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in
> the
> > > past
> > > > > > that
> > > > > > > > > > > connectors
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > actively developed components
> > > > because
> > > > > > > they
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > external system and with
> Flink.
> > > > Given
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> > > improving
> > > > > and
> > > > > > > > > > evolving
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can
> achieve
> > > > this.
> > > > > > > Hence,
> > > > > > > > > +1
> > > > > > > > > > > for
> > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03
> AM
> > > > Sijie
> > > > > > Guo
> > > > > > > <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main driver
> > > behind
> > > > > > > > > FLINK-9641
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168
> was
> > > > > created
> > > > > > > for
> > > > > > > > > > > bringing
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > integration
> > > > was
> > > > > > done
> > > > > > > > > with
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago
> and
> > we
> > > > > ended
> > > > > > > up
> > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to
> use
> > > > Flink
> > > > > > to
> > > > > > > > > > process
> > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > >
> > > > > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty
> > simple
> > > > and
> > > > > > > there
> > > > > > > > > is
> > > > > > > > > > no
> > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have
> > made a
> > > > lot
> > > > > > of
> > > > > > > > > > changes
> > > > > > > > > > > in
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > first-class
> > > > > > > citizen in
> > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > >> other computing engines for
> > > > > processing
> > > > > > > > > Pulsar
> > > > > > > > > > > > event
> > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how to
> > > > > integrate
> > > > > > > with
> > > > > > > > > > Flink
> > > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > >> reimplement the pulsar-flink
> > > > > > connectors
> > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > >> bring table API and catalog
> > API
> > > as
> > > > > the
> > > > > > > > > > > first-class
> > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > >> integration. With that being
> > > said,
> > > > > in
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar as a
> > > flink
> > > > > > > catalog
> > > > > > > > > and
> > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> This is an example about how
> > to
> > > > use
> > > > > > > Pulsar
> > > > > > > > > as
> > > > > > > > > > a
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a
> blog
> > > post
> > > > > > > > > explaining
> > > > > > > > > > > why
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9 and
> > > what
> > > > > are
> > > > > > > the
> > > > > > > > > > > changes
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not
> just
> > a
> > > > > simple
> > > > > > > data
> > > > > > > > > > sink
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> > > integrated
> > > > > > > streaming
> > > > > > > > > > > data
> > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > schema/catalog
> > > > > > and
> > > > > > > > > > state).
> > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a
> great
> > > > > > streaming
> > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified
> data
> > > > > > > processing.
> > > > > > > > > > Since
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> integration
> > to
> > > > > Flink
> > > > > > > here,
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and evolve
> > the
> > > > > > > integration
> > > > > > > > > > > with
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and
> Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit
> more
> > > > > > background
> > > > > > > > > about
> > > > > > > > > > > the
> > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know
> what
> > > are
> > > > > your
> > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at 11:54
> > AM
> > > > Yun
> > > > > > > Tang <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar
> becomes
> > > > more
> > > > > > and
> > > > > > > more
> > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > >>> to see more people willing
> to
> > > > > > > contribute to
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > discussion,
> > > > > would
> > > > > > > you
> > > > > > > > > > please
> > > > > > > > > > > > > give
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > >>> the relationship between
> this
> > > > > thread
> > > > > > to
> > > > > > > > > > current
> > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > > connector?
> > > > > > > Will the
> > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> > different
> > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September 3,
> > > 2019
> > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email] <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS]
> Contribute
> > > > > Pulsar
> > > > > > > Flink
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > > > discussion
> > > > > > of
> > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to
> Flink.
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction to
> > > Apache
> > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > > multi-tenant,
> > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> > > Pulsar
> > > > > > > includes
> > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > >>> native support for multiple
> > > > > clusters
> > > > > > > in a
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication of
> > > > > messages
> > > > > > > across
> > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> > > seamless
> > > > > > > > > scalability
> > > > > > > > > > to
> > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> > delivery
> > > > > with
> > > > > > > > > > persistent
> > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > BookKeeper.
> > > > > > > Nowadays,
> > > > > > > > > > Pulsar
> > > > > > > > > > > > has
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > >>> more and more companies[2].
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar
> Flink
> > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink connector
> we
> > > are
> > > > > > > planning
> > > > > > > > > to
> > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar
> 2.4.0.
> > > The
> > > > > > main
> > > > > > > > > > features
> > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> > source
> > > > with
> > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results to
> > > > Pulsar
> > > > > > with
> > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > > exactly-once
> > > > > as
> > > > > > > well
> > > > > > > > > > when
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > >>> transaction features ready
> in
> > > its
> > > > > > 2.5.0
> > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new
> Table
> > > API
> > > > > Type
> > > > > > > > > system
> > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > >>> automatically (de)serialize
> > > > > messages
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > help
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink new
> > > > Catalog
> > > > > > API
> > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> > tables
> > > in
> > > > > > > Table API
> > > > > > > > > > as
> > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Becket Qin]

Hi Sijie,

Yes, we will have to support existing old connectors and new connectors in
parallel for a while. We have to take that maintenance overhead because
existing connectors have been used by the users for a long time. I guess It
may take at least a year for us to fully remove the old connectors.

Process wise, we can do the same for Pulsar connector. But I am not sure if
we want to have the same burden on Pulsar connector, and I would like to
understand the benefit of doing that.

For users, the benefit of having the old Pulsar connector checked in seems
limited because 1) that code base will be immediately deprecated in the
next release in 3-4 months; 2) users can always use it even if it is not in
the Flink code base. Admittedly it is not as convenient as having it in
Flink code base, but doesn't seem super either. And after 3-4 months, users
can just use the new connector in Flink repo.

For Flink developers, the old connector code base is not something that we
want to evolve later. Instead, these code will be deprecated and
removed. So why do we want to get a beta version out to attract people to
use something we don't want to maintain?

Thanks,

Jiangjie (Becket) Qin



On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]> wrote:

> Thanks everyone here. Sorry for jumping into the discussion here.
>
> I am not very familiar about the deprecation process in Flink. If I
> misunderstood the process, please fix me.
>
> As far as I understand, FLIP-27 is introducing a new unified API for
> connectors. After it introduces the new API
> and before moving all the existing connectors from old API to new API, both
> old ApI and new API will co-exist
> for a while until Flink moves all existing connectors to new API. So the
> Pulsar connector (using old API) can
> follow the deprecation process with other connector using old API and the
> deprecation of old API, no?
>
> If that's the case, I think contributing the current connector back to
> Flink rather than maintaining it outside Flink
> would provide a bit more benefits. We can deprecate the existing
> streamnative/pulsar-flink repo and point the users
> to use the connector in Flink repo. So all the review processes will happen
> within Flink for both old connector and
> new connector. It also reduces the confusions for the users as the
> documentation and code base happen in one place.
>
> Thoughts?
>
> - Sijie
>
>
>
>
> On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <[hidden email]> wrote:
>
> > Thanks for the explanation, Stephan. I have a few questions / thoughts.
> >
> > So that means we will remove the old connector without a major version
> > bump, is that correct?
> >
> > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
> > always work because we saw some dependency class collisions in the past.
> To
> > make it safe we may have to maintain the old code for one more release.
> >
> > To be honest I am still wondering if we have to put the old connector in
> > Flink repo. if we check in the old connector to Flink. We will end up in
> > the following situation:
> > 1. Old connector in streamnative/pulsar-flink repo.
> > 2. Old connector in Flink Repo, which may be different from the one in
> > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> > 3. New connector in Flink Repo.
> >
> > We need to think about how to make the users in each case happy.
> > - For users of (1), I assume Sijie and Yijie will have to maintain the
> code
> > a bit longer for its own compatibility even after we have (2). In that
> > case, bugs found in old connector may or may not need to be fixed in both
> > Flink and the streamnative/pulsar-flink repo.
> > - For users of (2), will we provide bug fixes? If we do, it will be a
> > little awkward because those bug fixes will be immediately deprecated in
> > 1.11, and removed in 1.12. So we are essentially asking users to migrate
> > away from the bug fix. After Flink 1.12, users may still have to switch
> to
> > use (3) due to the potential dependency class conflicts mentioned above.
> > - Users of (3) have a much easier life and don't need to worry too much.
> >
> > The above story seems a little complicated to tell. I think it will be
> much
> > easier to not have (2) at all.
> > 1. Old connector in streamnative/pulsar-flink repo.
> > 3. New connector in Flink Repo.
> >
> > - Old connector will only be maintained in streamnative/pulsar-flink repo
> > until it is fully deprecated. Users can always use the existing Pulsar
> > connector in that repo.
> > - New connector will be in Flink repo and maintained like the other
> > connectors.
> >
> > This seems much simpler for users understand and they will not be blocked
> > from using the old connector. If the concern is about the quality of the
> > connector in streamnative/pulsar-flink repo, is it enough for us just to
> > review the code in streamnative/pulsar-flink connector to make sure it
> > looks good from Flink's perspective?
> >
> > What do you think?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]> wrote:
> >
> > > My take would be the following:
> > >
> > >   - If we merge the connector now and replace it with a FLIP-27 version
> > > before the 1.10 release, then we need no deprecation process
> > >   - If we don't manage to replace it with a FLIP-27 version before the
> > 1.10
> > > release, than it is good that we have the other version, so no users
> get
> > > blocked.
> > >
> > > In the latter case we can see how we want to do it. Immediate removal
> of
> > > the old version or deprecation label and keeping it for one more
> release.
> > > Given that you should be able to use a Flink 1.10 connector with Flink
> > 1.11
> > > as well (stable public APIs) there is also a workaround if you need an
> > old
> > > connector in a newer version. So immediate removal might even be
> > feasible.
> > >
> > >
> > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <[hidden email]>
> > wrote:
> > >
> > > > Hi Stephan,
> > > >
> > > > Thanks for the clarification. I completely agree with you and Thomas
> on
> > > the
> > > > process of adding connectors to Flink repo. However, I am wondering
> > what
> > > is
> > > > the deprecation process? Given the main concern here was that we may
> > have
> > > > to maintain two Pulsar connector code bases until the old one is
> > removed
> > > > from the repo, it would be good to know how long we have to do that.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]>
> wrote:
> > > >
> > > > > Some quick thoughts on the connector contribution process. I
> > basically
> > > > > reiterate here what Thomas mentioned in another thread about the
> > > Kinesis
> > > > > connector.
> > > > >
> > > > > For connectors, we should favor a low-overhead contribution
> process,
> > > and
> > > > > accept user code and changes more readily than in the core system.
> > > > > That is because connectors have both a big variety of scenarios
> they
> > > get
> > > > > used in (only through use and many small contributions do they
> become
> > > > > really useful over time) and at the same time, and committers do
> not
> > > use
> > > > > the connector themselves and usually cannot foresee too well what
> is
> > > > > needed.
> > > > >
> > > > > Further more, a missing connector (or connector feature) is often a
> > > > bigger
> > > > > show stopper for users than a missing API or system feature.
> > > > >
> > > > > Along these lines of thougt, the conclusion would be to take the
> > Pulsar
> > > > > connector now, focus the review on legal/dependencies/rough code
> > style
> > > > and
> > > > > conventions, label it as "beta" (in the sense of "new code" that is
> > > "not
> > > > > yet tested through longer use") and go ahead. And then evolve it
> > > quickly
> > > > > without putting formal blockers in the way, meaning also adding a
> new
> > > > FLIP
> > > > > 27 version when it is there.
> > > > >
> > > > > Best,
> > > > > Stephan
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hi Yijie,
> > > > > >
> > > > > > Could you please follow the FLIP process to start a new FLIP
> > > > [DISCUSSION]
> > > > > > thread in the mailing list?
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > >
> > > > > > I see two FLIP-69 discussion in the mailing list now. So there
> is a
> > > > FLIP
> > > > > > number collision. Can you change the FLIP number to 72?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > > > Hi Yijie,
> > > > > > >
> > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > Would you mind enabling comments/suggestions on the google doc
> > > link?
> > > > > This
> > > > > > > way the contributors from the community can comment on the doc.
> > > > > > >
> > > > > > > Best,
> > > > > > > Rong
> > > > > > >
> > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > > [hidden email]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > I've drafted a FLIP that describes the current design of the
> > > Pulsar
> > > > > > > > connector:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > >
> > > > > > > > Please take a look and let me know what you think.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yijie
> > > > > > > >
> > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Sorry for joining the discussion late and thanks Yijie &
> > Sijie
> > > > for
> > > > > > > > driving
> > > > > > > > > the discussion.
> > > > > > > > > I also think the Pulsar connector would be a very valuable
> > > > addition
> > > > > > to
> > > > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > > > >
> > > > > > > > > Regarding the timeline, I also share concerns with Becket
> on
> > > the
> > > > > > > > > relationship between the new Pulsar connector and FLIP-27.
> > > > > > > > > There's also another discussion just started by Stephan on
> > > > dropping
> > > > > > > Kafka
> > > > > > > > > 9/10 support on next Flink release [1].  Although the
> > situation
> > > > is
> > > > > > > > somewhat
> > > > > > > > > different, and Kafka 9/10 connector has been in Flink for
> > > almost
> > > > > 3-4
> > > > > > > > years,
> > > > > > > > > based on the discussion I am not sure if a major version
> > > release
> > > > > is a
> > > > > > > > > requirement for removing old connector supports.
> > > > > > > > >
> > > > > > > > > I think there shouldn't be a blocker if we agree the old
> > > > connector
> > > > > > will
> > > > > > > > be
> > > > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> > > Stephan
> > > > > > > stated,
> > > > > > > > it
> > > > > > > > > is easier to contribute the source sooner and adjust it
> > later.
> > > > > > > > > We should also ensure we clearly communicate the message:
> for
> > > > > > example,
> > > > > > > > > putting an experimental flag on the pre-FLIP27 connector
> page
> > > of
> > > > > the
> > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rong
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Technically speaking, removing the old connector code is
> a
> > > > > > backwards
> > > > > > > > > > incompatible change which requires a major version bump,
> > i.e.
> > > > > Flink
> > > > > > > > 2.x.
> > > > > > > > > > Given that we don't have a clear plan on when to have the
> > > next
> > > > > > major
> > > > > > > > > > version release, it seems unclear how long the old
> > connector
> > > > code
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > there if we check it in right now. Or will we remove the
> > old
> > > > > > > connector
> > > > > > > > > > without a major version bump? In any case, it sounds not
> > > quite
> > > > > user
> > > > > > > > > > friendly to the those who might use the old Pulsar
> > > connector. I
> > > > > am
> > > > > > > not
> > > > > > > > sure
> > > > > > > > > > if it is worth these potential problems in order to have
> > the
> > > > > Pulsar
> > > > > > > > source
> > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Agreed, if we check in the old code, we should make it
> > > clear
> > > > > that
> > > > > > > it
> > > > > > > > will
> > > > > > > > > > > be removed as soon as the FLIP-27 based version of the
> > > > > connector
> > > > > > is
> > > > > > > > > > there.
> > > > > > > > > > > We should not commit to maintaining the old version,
> that
> > > > would
> > > > > > be
> > > > > > > > indeed
> > > > > > > > > > > too much overhead.
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, the overhead would just be review capacity. In
> > > fact, I
> > > > > am
> > > > > > > not
> > > > > > > > > > > worrying
> > > > > > > > > > > > too much about the review capacity. That is just a
> one
> > > time
> > > > > > cost.
> > > > > > > > My
> > > > > > > > > > > > concern is mainly about the long term burden. Assume
> we
> > > > have
> > > > > > new
> > > > > > > > source
> > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > > connectors
> > > > in
> > > > > > old
> > > > > > > > > > > > interface. Later on if we migrate Pulsar to new
> source
> > > > > > interface,
> > > > > > > > the
> > > > > > > > > > old
> > > > > > > > > > > > Pulsar connector might be deprecated almost
> immediately
> > > > after
> > > > > > > > checked
> > > > > > > > > > in,
> > > > > > > > > > > > but we may still have to maintain two code bases. For
> > the
> > > > > > > existing
> > > > > > > > > > > > connectors, we have to do that anyways. But it would
> be
> > > > good
> > > > > to
> > > > > > > > avoid
> > > > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Nice to see this lively discussion about the Pulsar
> > > > > > connector.
> > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > >
> > > > > > > > > > > > > ## Contribute to Flink or maintain as a community
> > > package
> > > > > > > > > > > > >
> > > > > > > > > > > > > Looks like the discussion is more going towards
> > > > > > contribution. I
> > > > > > > > think
> > > > > > > > > > > > that
> > > > > > > > > > > > > is good, especially if we think that we want to
> > build a
> > > > > > > similarly
> > > > > > > > > > deep
> > > > > > > > > > > > > integration with Pulsar as we have for example with
> > > > Kafka.
> > > > > > The
> > > > > > > > > > > connector
> > > > > > > > > > > > > already looks like a more thorough connector than
> > many
> > > > > others
> > > > > > > we
> > > > > > > > have
> > > > > > > > > > > in
> > > > > > > > > > > > > the repository.
> > > > > > > > > > > > >
> > > > > > > > > > > > > With either a repo split, or the new build system,
> I
> > > hope
> > > > > > that
> > > > > > > > the
> > > > > > > > > > > build
> > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > >
> > > > > > > > > > > > > Becket offered some help already, I can also help a
> > > bit.
> > > > I
> > > > > > hope
> > > > > > > > that
> > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > >
> > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some
> > PoC-ing
> > > > > > > > progress, but
> > > > > > > > > > > > will
> > > > > > > > > > > > > take 2 more months, I would estimate, before it is
> > > fully
> > > > > > > > available.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we want to be on the safe side with the
> > > contribution,
> > > > we
> > > > > > > > should
> > > > > > > > > > > > > contribute the source sooner and adjust it later.
> > That
> > > > > would
> > > > > > > also
> > > > > > > > > > help
> > > > > > > > > > > us
> > > > > > > > > > > > > in case things get crazy towards the 1.10 feature
> > > freeze
> > > > > and
> > > > > > it
> > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > > What would be the overhead of contributing now?
> Given
> > > > that
> > > > > > the
> > > > > > > > code
> > > > > > > > > > is
> > > > > > > > > > > > > already there, it looks like it would be only
> review
> > > > > > capacity,
> > > > > > > > right?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Stephan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your attention and the promotion of
> this
> > > > work.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We will prepare a FLIP as soon as possible for
> more
> > > > > > specific
> > > > > > > > > > > > discussions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For FLIP-27, it seems that we have not reached a
> > > > > consensus.
> > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > I will explain all the functionalities of the
> > > existing
> > > > > > > > connector in
> > > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog) to
> > > > > continue
> > > > > > > our
> > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> > > connector
> > > > > in
> > > > > > > > 1.10,
> > > > > > > > > > how
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to
> Flink
> > > main
> > > > > > repo
> > > > > > > > as it
> > > > > > > > > > > is a
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away
> as
> > > > there
> > > > > is
> > > > > > > no
> > > > > > > > > > change
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10 is
> > > going
> > > > to
> > > > > > be
> > > > > > > > code
> > > > > > > > > > > > freeze
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > > > development
> > > > > > > and
> > > > > > > > > > > review
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by
> late
> > > > Oct.
> > > > > > > There
> > > > > > > > are
> > > > > > > > > > > > still
> > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > weeks. Personally I think it is doable. If
> > FLIP-27
> > > is
> > > > > not
> > > > > > > > ready
> > > > > > > > > > by
> > > > > > > > > > > > late
> > > > > > > > > > > > > > > Oct, we can review and check in Pulsar
> connector
> > > with
> > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > interface. This means we will have Pulsar
> > connector
> > > > in
> > > > > > > Flink
> > > > > > > > > > 1.10,
> > > > > > > > > > > > > either
> > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Because we are going to have Pulsar sink and
> > source
> > > > > > checked
> > > > > > > > in
> > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > it might make sense to have two FLIPs, one for
> > > Pulsar
> > > > > > sink
> > > > > > > > and
> > > > > > > > > > > > another
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > Pulsar source. And we can start the work on
> > Pulsar
> > > > sink
> > > > > > > right
> > > > > > > > > > away.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What's the take from Flink community? Shall
> we
> > > wait
> > > > > for
> > > > > > > > FLIP-27
> > > > > > > > > > > or
> > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > proceed to next steps? And what the next
> steps
> > > are?
> > > > > :-)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think having a Pulsar connector in Flink
> > can
> > > > be a
> > > > > > > good
> > > > > > > > > > mutual
> > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Another perspective is that Pulsar
> connector
> > is
> > > > the
> > > > > > 1st
> > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > > management
> > > > > > system
> > > > > > > > and
> > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > It'll be cool to see how the integration
> > turns
> > > > out
> > > > > > and
> > > > > > > > > > whether
> > > > > > > > > > > we
> > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> > > currently
> > > > in
> > > > > > > > Beta, to
> > > > > > > > > > > > cater
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor of
> > > > merging
> > > > > > > > Pulsar
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs,
> e.g.
> > > > maybe
> > > > > > one
> > > > > > > > for
> > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > source/sink functionalities and another for
> > > > schema
> > > > > > and
> > > > > > > > > > catalog
> > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for
> FLIP-27.
> > > But
> > > > I
> > > > > > > don't
> > > > > > > > > > think
> > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > should be a blocker in cases where it
> cannot
> > > make
> > > > > its
> > > > > > > way
> > > > > > > > > > into
> > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time for
> > > > > > committers
> > > > > > > to
> > > > > > > > > > > review
> > > > > > > > > > > > or
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > > > interfaces.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket Qin <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > You are right. It all depends on when the
> > new
> > > > > > source
> > > > > > > > > > > interface
> > > > > > > > > > > > is
> > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > be ready. Personally I think it would be
> > > there
> > > > in
> > > > > > > > about a
> > > > > > > > > > > month
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > I could be too optimistic. It would also
> be
> > > > good
> > > > > to
> > > > > > > > hear
> > > > > > > > > > what
> > > > > > > > > > > > do
> > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > and Stephan think as they are also
> involved
> > > in
> > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > In general I think we should have Pulsar
> > > > > connector
> > > > > > in
> > > > > > > > Flink
> > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > preferably with the new source interface.
> > We
> > > > can
> > > > > > also
> > > > > > > > check
> > > > > > > > > > > it
> > > > > > > > > > > > in
> > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > with old source interface, but I suspect
> > few
> > > > > users
> > > > > > > > will use
> > > > > > > > > > > it
> > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > next official release. Therefore, it
> seems
> > > > > > reasonable
> > > > > > > > to
> > > > > > > > > > > wait a
> > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > to see whether we can jump to the new
> > source
> > > > > > > > interface. As
> > > > > > > > > > > long
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a little
> > bit
> > > > > > doesn't
> > > > > > > > seem
> > > > > > > > > > to
> > > > > > > > > > > > hurt
> > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till
> > Rohrmann
> > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'm wondering what the problem would be
> > if
> > > we
> > > > > > > > committed
> > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > connector before the new source
> interface
> > > is
> > > > > > ready.
> > > > > > > > If I
> > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > correctly, then we need to support the
> > old
> > > > > source
> > > > > > > > > > interface
> > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > existing connectors. By checking it in
> > > early
> > > > I
> > > > > > > could
> > > > > > > > see
> > > > > > > > > > > the
> > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > our users could start using the
> connector
> > > > > > earlier.
> > > > > > > > > > > Moreover,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration is
> > > being
> > > > > > > delayed
> > > > > > > > in
> > > > > > > > > > > case
> > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > source interface should be delayed. The
> > > only
> > > > > > > > downside I
> > > > > > > > > > see
> > > > > > > > > > > > is
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > review effort and potential fixes which
> > > might
> > > > > be
> > > > > > > > > > irrelevant
> > > > > > > > > > > > for
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > source interface implementation. I
> guess
> > it
> > > > > > mainly
> > > > > > > > > > depends
> > > > > > > > > > > on
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > we are when the new source interface
> will
> > > be
> > > > > > ready.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket
> > Qin <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Just want to have some update on
> > FLIP-27.
> > > > > > > Although
> > > > > > > > the
> > > > > > > > > > > FLIP
> > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > discussion thread has been quiet for
> > some
> > > > > > time, a
> > > > > > > > few
> > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > contributors in Flink community were
> > > > actually
> > > > > > > > > > prototyping
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > We have made some good progress there
> > but
> > > > > want
> > > > > > to
> > > > > > > > > > update
> > > > > > > > > > > > the
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > after the entire thing is verified to
> > > work
> > > > in
> > > > > > > case
> > > > > > > > > > there
> > > > > > > > > > > > are
> > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > minute surprise in the
> implementation.
> > I
> > > > > don't
> > > > > > > > have an
> > > > > > > > > > > > exact
> > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > guess it is going to be within a
> month
> > or
> > > > so.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I am happy to review the current
> Flink
> > > > Pulsar
> > > > > > > > connector
> > > > > > > > > > > and
> > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to
> > avoid
> > > > the
> > > > > > > case
> > > > > > > > that
> > > > > > > > > > > we
> > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Pulsar connector with some review
> > efforts
> > > > and
> > > > > > > > shortly
> > > > > > > > > > > after
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM Yijie
> > > Shen <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> > > connector
> > > > > has
> > > > > > > > always
> > > > > > > > > > > been
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > users with the latest features of
> > both
> > > > > > systems
> > > > > > > as
> > > > > > > > > > soon
> > > > > > > > > > > as
> > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > propose to contribute the connector
> > to
> > > > > Flink
> > > > > > > and
> > > > > > > > hope
> > > > > > > > > > > to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > suggestions and feedback from Flink
> > > > experts
> > > > > > to
> > > > > > > > ensure
> > > > > > > > > > > the
> > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its
> existence
> > > at
> > > > > the
> > > > > > > > > > beginning
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > the connector implementation based
> on
> > > > Flink
> > > > > > > 1.9;
> > > > > > > > we
> > > > > > > > > > > also
> > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > build a connector that supports
> both
> > > > batch
> > > > > > and
> > > > > > > > stream
> > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > However, it has been inactive for
> > some
> > > > > time,
> > > > > > so
> > > > > > > > we
> > > > > > > > > > > > decided
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > a connector with most of the new
> > > > features,
> > > > > > such
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > and the new catalog API first. We
> > will
> > > > pay
> > > > > > > > attention
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> > incorporate
> > > it
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Regarding the test status of the
> > > > connector,
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > > following
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> repository
> > > and
> > > > > > aimed
> > > > > > > to
> > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > tests as we could. We are also
> happy
> > to
> > > > > hear
> > > > > > > > > > > suggestions
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > supervision from the Flink
> community
> > to
> > > > > > improve
> > > > > > > > the
> > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > performance of the connector
> > > > continuously.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM
> Sijie
> > > Guo
> > > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the comments
> > and
> > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It seems to me that the main
> > question
> > > > > here
> > > > > > is
> > > > > > > > > > about -
> > > > > > > > > > > > > "how
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community maintain the
> connector?".
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from
> myself.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to host
> > this
> > > > > > > > integration
> > > > > > > > > > is
> > > > > > > > > > > > kind
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > here. I believe there can be many
> > > ways
> > > > to
> > > > > > > > achieve
> > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > As part of the contribution, what
> > we
> > > > are
> > > > > > > > looking
> > > > > > > > > > for
> > > > > > > > > > > > here
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > communities can build the
> > > collaboration
> > > > > > > > > > relationship
> > > > > > > > > > > on
> > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > the integration between Pulsar
> and
> > > > Flink.
> > > > > > > Even
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > try
> > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > > community.
> > > > We
> > > > > > are
> > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > facing the fact that we have less
> > > > > > experiences
> > > > > > > > in
> > > > > > > > > > > Flink
> > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community. In order to make sure
> we
> > > > > > maintain
> > > > > > > > and
> > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > > integration
> > > > > to
> > > > > > > the
> > > > > > > > > > users
> > > > > > > > > > > > who
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > technologies, we need some help
> > from
> > > > the
> > > > > > > > experts
> > > > > > > > > > from
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 2) We have been following FLIP-27
> > > for a
> > > > > > > while.
> > > > > > > > > > > > Originally
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > of contributing the connectors
> back
> > > > after
> > > > > > > > > > integrating
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27.
> But
> > we
> > > > > > decided
> > > > > > > > to
> > > > > > > > > > > > initiate
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > as early as possible. Because we
> > > > believe
> > > > > > > there
> > > > > > > > are
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > it now rather than later. As part
> > of
> > > > > > > > contribution,
> > > > > > > > > > it
> > > > > > > > > > > > can
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community understand more about
> > > Pulsar
> > > > > and
> > > > > > > the
> > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink
> > community
> > > > > > verify
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog
> > API).
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM
> > Becket
> > > > > Qin <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > > > contributing
> > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > In general, I think having
> Pulsar
> > > > > > connector
> > > > > > > > with
> > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink. So
> I
> > am
> > > > > happy
> > > > > > > the
> > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like to
> > > > provide
> > > > > > > some
> > > > > > > > > > > context
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > the Flink connectors ecosystem.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > The current way Flink maintains
> > its
> > > > > > > > connector has
> > > > > > > > > > > hit
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > With more and more connectors
> > > coming
> > > > > into
> > > > > > > > Flink
> > > > > > > > > > > repo,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > problems such as long build and
> > > > testing
> > > > > > > > time. To
> > > > > > > > > > > > > address
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > have attempted to do the
> > following:
> > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors
> into
> > a
> > > > > > separate
> > > > > > > > > > > > repository.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > on hold due to potential
> solution
> > > to
> > > > > > > shorten
> > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors to
> > stay
> > > > as
> > > > > > > > ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > > > functionality
> > > > > > > and
> > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> > > > Ecosystem
> > > > > > > > project
> > > > > > > > > > > > website
> > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > through some final approval
> > > process.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it
> would
> > > be
> > > > > > great
> > > > > > > to
> > > > > > > > > > first
> > > > > > > > > > > > see
> > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an
> ecosystem
> > > > > project
> > > > > > > with
> > > > > > > > > > great
> > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink
> Pulsar
> > > > > > connector
> > > > > > > > is
> > > > > > > > > > > tested
> > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > we can learn something to
> > maintain
> > > it
> > > > > as
> > > > > > an
> > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If
> the
> > > > > quality
> > > > > > > as
> > > > > > > > an
> > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well
> > adopt
> > > it
> > > > > > into
> > > > > > > > the
> > > > > > > > > > main
> > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort is
> > > > FLIP-27
> > > > > > > where
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > architecture
> > > > and
> > > > > > > > > > interface.
> > > > > > > > > > > > This
> > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore timing
> > > wise,
> > > > if
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > going
> > > > > > > > > > > > to
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> > > > wondering
> > > > > if
> > > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > > hold
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector adapt
> to
> > > the
> > > > > new
> > > > > > > > > > interface
> > > > > > > > > > > > to
> > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32 PM
> > > > Chesnay
> > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we may
> > end
> > > > up
> > > > > > > > repeating
> > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > There were already 2 attempts
> > at
> > > > > > > > contributing a
> > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > of which failed because no
> > > > committer
> > > > > > was
> > > > > > > > > > getting
> > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > contributor opening a
> dedicated
> > > > > > > discussion
> > > > > > > > > > thread
> > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting
> several
> > > +1's
> > > > > > from
> > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > We should really make sure
> that
> > > if
> > > > we
> > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > contribution it will actually
> > get
> > > > the
> > > > > > > > attention
> > > > > > > > > > > it
> > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> > > recommend
> > > > > > > > maintaining
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it
> from
> > > the
> > > > > > > > > > documentation
> > > > > > > > > > > > to
> > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> > > sharing
> > > > > > > > artifacts
> > > > > > > > > > > among
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?),
> > this
> > > > may
> > > > > > be a
> > > > > > > > > > better
> > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> > > Rohrmann
> > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting
> > this
> > > > > > > discussion
> > > > > > > > > > > Yijie.
> > > > > > > > > > > > I
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> > > > valuable
> > > > > > > > addition
> > > > > > > > > > > since
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> > > further
> > > > > > > expand
> > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > from a project perspective
> it
> > > > makes
> > > > > > > > sense for
> > > > > > > > > > > me
> > > > > > > > > > > > to
> > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > My main concern/question is
> > how
> > > > can
> > > > > > the
> > > > > > > > Flink
> > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen in
> > the
> > > > past
> > > > > > > that
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > actively developed
> components
> > > > > because
> > > > > > > > they
> > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > external system and with
> > Flink.
> > > > > Given
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> > > > improving
> > > > > > and
> > > > > > > > > > > evolving
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can
> > achieve
> > > > > this.
> > > > > > > > Hence,
> > > > > > > > > > +1
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 2:03
> > AM
> > > > > Sijie
> > > > > > > Guo
> > > > > > > > <
> > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main
> driver
> > > > behind
> > > > > > > > > > FLINK-9641
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > >> add more context on this.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and FLINK-9168
> > was
> > > > > > created
> > > > > > > > for
> > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > > integration
> > > > > was
> > > > > > > done
> > > > > > > > > > with
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year ago
> > and
> > > we
> > > > > > ended
> > > > > > > > up
> > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users to
> > use
> > > > > Flink
> > > > > > > to
> > > > > > > > > > > process
> > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > >
> https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is pretty
> > > simple
> > > > > and
> > > > > > > > there
> > > > > > > > > > is
> > > > > > > > > > > no
> > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we have
> > > made a
> > > > > lot
> > > > > > > of
> > > > > > > > > > > changes
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > > first-class
> > > > > > > > citizen in
> > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > >> other computing engines
> for
> > > > > > processing
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how
> to
> > > > > > integrate
> > > > > > > > with
> > > > > > > > > > > Flink
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> pulsar-flink
> > > > > > > connectors
> > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and
> catalog
> > > API
> > > > as
> > > > > > the
> > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that
> being
> > > > said,
> > > > > > in
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar
> as a
> > > > flink
> > > > > > > > catalog
> > > > > > > > > > and
> > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> This is an example about
> how
> > > to
> > > > > use
> > > > > > > > Pulsar
> > > > > > > > > > as
> > > > > > > > > > > a
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a
> > blog
> > > > post
> > > > > > > > > > explaining
> > > > > > > > > > > > why
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9
> and
> > > > what
> > > > > > are
> > > > > > > > the
> > > > > > > > > > > > changes
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not
> > just
> > > a
> > > > > > simple
> > > > > > > > data
> > > > > > > > > > > sink
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> > > > integrated
> > > > > > > > streaming
> > > > > > > > > > > > data
> > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > > schema/catalog
> > > > > > > and
> > > > > > > > > > > state).
> > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a
> > great
> > > > > > > streaming
> > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified
> > data
> > > > > > > > processing.
> > > > > > > > > > > Since
> > > > > > > > > > > > we
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> > integration
> > > to
> > > > > > Flink
> > > > > > > > here,
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and
> evolve
> > > the
> > > > > > > > integration
> > > > > > > > > > > > with
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and
> > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit
> > more
> > > > > > > background
> > > > > > > > > > about
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know
> > what
> > > > are
> > > > > > your
> > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at
> 11:54
> > > AM
> > > > > Yun
> > > > > > > > Tang <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar
> > becomes
> > > > > more
> > > > > > > and
> > > > > > > > more
> > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people
> willing
> > to
> > > > > > > > contribute to
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > > discussion,
> > > > > > would
> > > > > > > > you
> > > > > > > > > > > please
> > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship between
> > this
> > > > > > thread
> > > > > > > to
> > > > > > > > > > > current
> > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > > > connector?
> > > > > > > > Will the
> > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> > > different
> > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday, September
> 3,
> > > > 2019
> > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > >>> To: [hidden email]
> <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS]
> > Contribute
> > > > > > Pulsar
> > > > > > > > Flink
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open the
> > > > > discussion
> > > > > > > of
> > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to
> > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction
> to
> > > > Apache
> > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > > > multi-tenant,
> > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging system.
> > > > Pulsar
> > > > > > > > includes
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > >>> native support for
> multiple
> > > > > > clusters
> > > > > > > > in a
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > >>> seamless geo-replication
> of
> > > > > > messages
> > > > > > > > across
> > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> > > > seamless
> > > > > > > > > > scalability
> > > > > > > > > > > to
> > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> > > delivery
> > > > > > with
> > > > > > > > > > > persistent
> > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > > BookKeeper.
> > > > > > > > Nowadays,
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar
> > Flink
> > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink
> connector
> > we
> > > > are
> > > > > > > > planning
> > > > > > > > > > to
> > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar
> > 2.4.0.
> > > > The
> > > > > > > main
> > > > > > > > > > > features
> > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> > > source
> > > > > with
> > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming results
> to
> > > > > Pulsar
> > > > > > > with
> > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > > > exactly-once
> > > > > > as
> > > > > > > > well
> > > > > > > > > > > when
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features
> ready
> > in
> > > > its
> > > > > > > 2.5.0
> > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new
> > Table
> > > > API
> > > > > > Type
> > > > > > > > > > system
> > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> (de)serialize
> > > > > > messages
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink
> new
> > > > > Catalog
> > > > > > > API
> > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> > > tables
> > > > in
> > > > > > > > Table API
> > > > > > > > > > > as
> > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Sijie Guo-2]

Thanks Becket.

I think it is better for the Flink community to judge the benefits of doing
this. I was trying to provide some views from outsiders.

Thanks,
Sijie

On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]> wrote:

> Hi Sijie,
>
> Yes, we will have to support existing old connectors and new connectors in
> parallel for a while. We have to take that maintenance overhead because
> existing connectors have been used by the users for a long time. I guess It
> may take at least a year for us to fully remove the old connectors.
>
> Process wise, we can do the same for Pulsar connector. But I am not sure if
> we want to have the same burden on Pulsar connector, and I would like to
> understand the benefit of doing that.
>
> For users, the benefit of having the old Pulsar connector checked in seems
> limited because 1) that code base will be immediately deprecated in the
> next release in 3-4 months; 2) users can always use it even if it is not in
> the Flink code base. Admittedly it is not as convenient as having it in
> Flink code base, but doesn't seem super either. And after 3-4 months, users
> can just use the new connector in Flink repo.
>
> For Flink developers, the old connector code base is not something that we
> want to evolve later. Instead, these code will be deprecated and
> removed. So why do we want to get a beta version out to attract people to
> use something we don't want to maintain?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]> wrote:
>
> > Thanks everyone here. Sorry for jumping into the discussion here.
> >
> > I am not very familiar about the deprecation process in Flink. If I
> > misunderstood the process, please fix me.
> >
> > As far as I understand, FLIP-27 is introducing a new unified API for
> > connectors. After it introduces the new API
> > and before moving all the existing connectors from old API to new API,
> both
> > old ApI and new API will co-exist
> > for a while until Flink moves all existing connectors to new API. So the
> > Pulsar connector (using old API) can
> > follow the deprecation process with other connector using old API and the
> > deprecation of old API, no?
> >
> > If that's the case, I think contributing the current connector back to
> > Flink rather than maintaining it outside Flink
> > would provide a bit more benefits. We can deprecate the existing
> > streamnative/pulsar-flink repo and point the users
> > to use the connector in Flink repo. So all the review processes will
> happen
> > within Flink for both old connector and
> > new connector. It also reduces the confusions for the users as the
> > documentation and code base happen in one place.
> >
> > Thoughts?
> >
> > - Sijie
> >
> >
> >
> >
> > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <[hidden email]>
> wrote:
> >
> > > Thanks for the explanation, Stephan. I have a few questions / thoughts.
> > >
> > > So that means we will remove the old connector without a major version
> > > bump, is that correct?
> > >
> > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors will
> > > always work because we saw some dependency class collisions in the
> past.
> > To
> > > make it safe we may have to maintain the old code for one more release.
> > >
> > > To be honest I am still wondering if we have to put the old connector
> in
> > > Flink repo. if we check in the old connector to Flink. We will end up
> in
> > > the following situation:
> > > 1. Old connector in streamnative/pulsar-flink repo.
> > > 2. Old connector in Flink Repo, which may be different from the one in
> > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> > > 3. New connector in Flink Repo.
> > >
> > > We need to think about how to make the users in each case happy.
> > > - For users of (1), I assume Sijie and Yijie will have to maintain the
> > code
> > > a bit longer for its own compatibility even after we have (2). In that
> > > case, bugs found in old connector may or may not need to be fixed in
> both
> > > Flink and the streamnative/pulsar-flink repo.
> > > - For users of (2), will we provide bug fixes? If we do, it will be a
> > > little awkward because those bug fixes will be immediately deprecated
> in
> > > 1.11, and removed in 1.12. So we are essentially asking users to
> migrate
> > > away from the bug fix. After Flink 1.12, users may still have to switch
> > to
> > > use (3) due to the potential dependency class conflicts mentioned
> above.
> > > - Users of (3) have a much easier life and don't need to worry too
> much.
> > >
> > > The above story seems a little complicated to tell. I think it will be
> > much
> > > easier to not have (2) at all.
> > > 1. Old connector in streamnative/pulsar-flink repo.
> > > 3. New connector in Flink Repo.
> > >
> > > - Old connector will only be maintained in streamnative/pulsar-flink
> repo
> > > until it is fully deprecated. Users can always use the existing Pulsar
> > > connector in that repo.
> > > - New connector will be in Flink repo and maintained like the other
> > > connectors.
> > >
> > > This seems much simpler for users understand and they will not be
> blocked
> > > from using the old connector. If the concern is about the quality of
> the
> > > connector in streamnative/pulsar-flink repo, is it enough for us just
> to
> > > review the code in streamnative/pulsar-flink connector to make sure it
> > > looks good from Flink's perspective?
> > >
> > > What do you think?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]> wrote:
> > >
> > > > My take would be the following:
> > > >
> > > >   - If we merge the connector now and replace it with a FLIP-27
> version
> > > > before the 1.10 release, then we need no deprecation process
> > > >   - If we don't manage to replace it with a FLIP-27 version before
> the
> > > 1.10
> > > > release, than it is good that we have the other version, so no users
> > get
> > > > blocked.
> > > >
> > > > In the latter case we can see how we want to do it. Immediate removal
> > of
> > > > the old version or deprecation label and keeping it for one more
> > release.
> > > > Given that you should be able to use a Flink 1.10 connector with
> Flink
> > > 1.11
> > > > as well (stable public APIs) there is also a workaround if you need
> an
> > > old
> > > > connector in a newer version. So immediate removal might even be
> > > feasible.
> > > >
> > > >
> > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi Stephan,
> > > > >
> > > > > Thanks for the clarification. I completely agree with you and
> Thomas
> > on
> > > > the
> > > > > process of adding connectors to Flink repo. However, I am wondering
> > > what
> > > > is
> > > > > the deprecation process? Given the main concern here was that we
> may
> > > have
> > > > > to maintain two Pulsar connector code bases until the old one is
> > > removed
> > > > > from the repo, it would be good to know how long we have to do
> that.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]>
> > wrote:
> > > > >
> > > > > > Some quick thoughts on the connector contribution process. I
> > > basically
> > > > > > reiterate here what Thomas mentioned in another thread about the
> > > > Kinesis
> > > > > > connector.
> > > > > >
> > > > > > For connectors, we should favor a low-overhead contribution
> > process,
> > > > and
> > > > > > accept user code and changes more readily than in the core
> system.
> > > > > > That is because connectors have both a big variety of scenarios
> > they
> > > > get
> > > > > > used in (only through use and many small contributions do they
> > become
> > > > > > really useful over time) and at the same time, and committers do
> > not
> > > > use
> > > > > > the connector themselves and usually cannot foresee too well what
> > is
> > > > > > needed.
> > > > > >
> > > > > > Further more, a missing connector (or connector feature) is
> often a
> > > > > bigger
> > > > > > show stopper for users than a missing API or system feature.
> > > > > >
> > > > > > Along these lines of thougt, the conclusion would be to take the
> > > Pulsar
> > > > > > connector now, focus the review on legal/dependencies/rough code
> > > style
> > > > > and
> > > > > > conventions, label it as "beta" (in the sense of "new code" that
> is
> > > > "not
> > > > > > yet tested through longer use") and go ahead. And then evolve it
> > > > quickly
> > > > > > without putting formal blockers in the way, meaning also adding a
> > new
> > > > > FLIP
> > > > > > 27 version when it is there.
> > > > > >
> > > > > > Best,
> > > > > > Stephan
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Yijie,
> > > > > > >
> > > > > > > Could you please follow the FLIP process to start a new FLIP
> > > > > [DISCUSSION]
> > > > > > > thread in the mailing list?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > > >
> > > > > > > I see two FLIP-69 discussion in the mailing list now. So there
> > is a
> > > > > FLIP
> > > > > > > number collision. Can you change the FLIP number to 72?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> [hidden email]>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yijie,
> > > > > > > >
> > > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > > Would you mind enabling comments/suggestions on the google
> doc
> > > > link?
> > > > > > This
> > > > > > > > way the contributors from the community can comment on the
> doc.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Rong
> > > > > > > >
> > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello everyone,
> > > > > > > > >
> > > > > > > > > I've drafted a FLIP that describes the current design of
> the
> > > > Pulsar
> > > > > > > > > connector:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > > >
> > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Yijie
> > > > > > > > >
> > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > Sorry for joining the discussion late and thanks Yijie &
> > > Sijie
> > > > > for
> > > > > > > > > driving
> > > > > > > > > > the discussion.
> > > > > > > > > > I also think the Pulsar connector would be a very
> valuable
> > > > > addition
> > > > > > > to
> > > > > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > > > > >
> > > > > > > > > > Regarding the timeline, I also share concerns with Becket
> > on
> > > > the
> > > > > > > > > > relationship between the new Pulsar connector and
> FLIP-27.
> > > > > > > > > > There's also another discussion just started by Stephan
> on
> > > > > dropping
> > > > > > > > Kafka
> > > > > > > > > > 9/10 support on next Flink release [1].  Although the
> > > situation
> > > > > is
> > > > > > > > > somewhat
> > > > > > > > > > different, and Kafka 9/10 connector has been in Flink for
> > > > almost
> > > > > > 3-4
> > > > > > > > > years,
> > > > > > > > > > based on the discussion I am not sure if a major version
> > > > release
> > > > > > is a
> > > > > > > > > > requirement for removing old connector supports.
> > > > > > > > > >
> > > > > > > > > > I think there shouldn't be a blocker if we agree the old
> > > > > connector
> > > > > > > will
> > > > > > > > > be
> > > > > > > > > > removed once FLIP-27 based Pulsar connector is there. As
> > > > Stephan
> > > > > > > > stated,
> > > > > > > > > it
> > > > > > > > > > is easier to contribute the source sooner and adjust it
> > > later.
> > > > > > > > > > We should also ensure we clearly communicate the message:
> > for
> > > > > > > example,
> > > > > > > > > > putting an experimental flag on the pre-FLIP27 connector
> > page
> > > > of
> > > > > > the
> > > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Rong
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Technically speaking, removing the old connector code
> is
> > a
> > > > > > > backwards
> > > > > > > > > > > incompatible change which requires a major version
> bump,
> > > i.e.
> > > > > > Flink
> > > > > > > > > 2.x.
> > > > > > > > > > > Given that we don't have a clear plan on when to have
> the
> > > > next
> > > > > > > major
> > > > > > > > > > > version release, it seems unclear how long the old
> > > connector
> > > > > code
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > there if we check it in right now. Or will we remove
> the
> > > old
> > > > > > > > connector
> > > > > > > > > > > without a major version bump? In any case, it sounds
> not
> > > > quite
> > > > > > user
> > > > > > > > > > > friendly to the those who might use the old Pulsar
> > > > connector. I
> > > > > > am
> > > > > > > > not
> > > > > > > > > sure
> > > > > > > > > > > if it is worth these potential problems in order to
> have
> > > the
> > > > > > Pulsar
> > > > > > > > > source
> > > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Agreed, if we check in the old code, we should make
> it
> > > > clear
> > > > > > that
> > > > > > > > it
> > > > > > > > > will
> > > > > > > > > > > > be removed as soon as the FLIP-27 based version of
> the
> > > > > > connector
> > > > > > > is
> > > > > > > > > > > there.
> > > > > > > > > > > > We should not commit to maintaining the old version,
> > that
> > > > > would
> > > > > > > be
> > > > > > > > > indeed
> > > > > > > > > > > > too much overhead.
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yes, the overhead would just be review capacity. In
> > > > fact, I
> > > > > > am
> > > > > > > > not
> > > > > > > > > > > > worrying
> > > > > > > > > > > > > too much about the review capacity. That is just a
> > one
> > > > time
> > > > > > > cost.
> > > > > > > > > My
> > > > > > > > > > > > > concern is mainly about the long term burden.
> Assume
> > we
> > > > > have
> > > > > > > new
> > > > > > > > > source
> > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > > > connectors
> > > > > in
> > > > > > > old
> > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new
> > source
> > > > > > > interface,
> > > > > > > > > the
> > > > > > > > > > > old
> > > > > > > > > > > > > Pulsar connector might be deprecated almost
> > immediately
> > > > > after
> > > > > > > > > checked
> > > > > > > > > > > in,
> > > > > > > > > > > > > but we may still have to maintain two code bases.
> For
> > > the
> > > > > > > > existing
> > > > > > > > > > > > > connectors, we have to do that anyways. But it
> would
> > be
> > > > > good
> > > > > > to
> > > > > > > > > avoid
> > > > > > > > > > > > > introducing a new connector with the same problem.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Nice to see this lively discussion about the
> Pulsar
> > > > > > > connector.
> > > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ## Contribute to Flink or maintain as a community
> > > > package
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looks like the discussion is more going towards
> > > > > > > contribution. I
> > > > > > > > > think
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > is good, especially if we think that we want to
> > > build a
> > > > > > > > similarly
> > > > > > > > > > > deep
> > > > > > > > > > > > > > integration with Pulsar as we have for example
> with
> > > > > Kafka.
> > > > > > > The
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > already looks like a more thorough connector than
> > > many
> > > > > > others
> > > > > > > > we
> > > > > > > > > have
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the repository.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With either a repo split, or the new build
> system,
> > I
> > > > hope
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > > build
> > > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Becket offered some help already, I can also
> help a
> > > > bit.
> > > > > I
> > > > > > > hope
> > > > > > > > > that
> > > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some
> > > PoC-ing
> > > > > > > > > progress, but
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > take 2 more months, I would estimate, before it
> is
> > > > fully
> > > > > > > > > available.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we want to be on the safe side with the
> > > > contribution,
> > > > > we
> > > > > > > > > should
> > > > > > > > > > > > > > contribute the source sooner and adjust it later.
> > > That
> > > > > > would
> > > > > > > > also
> > > > > > > > > > > help
> > > > > > > > > > > > us
> > > > > > > > > > > > > > in case things get crazy towards the 1.10 feature
> > > > freeze
> > > > > > and
> > > > > > > it
> > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > > > What would be the overhead of contributing now?
> > Given
> > > > > that
> > > > > > > the
> > > > > > > > > code
> > > > > > > > > > > is
> > > > > > > > > > > > > > already there, it looks like it would be only
> > review
> > > > > > > capacity,
> > > > > > > > > right?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your attention and the promotion of
> > this
> > > > > work.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible for
> > more
> > > > > > > specific
> > > > > > > > > > > > > discussions.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For FLIP-27, it seems that we have not reached
> a
> > > > > > consensus.
> > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > I will explain all the functionalities of the
> > > > existing
> > > > > > > > > connector in
> > > > > > > > > > > > > > > the FLIP (including Source, Sink, and Catalog)
> to
> > > > > > continue
> > > > > > > > our
> > > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> > > > connector
> > > > > > in
> > > > > > > > > 1.10,
> > > > > > > > > > > how
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to
> > Flink
> > > > main
> > > > > > > repo
> > > > > > > > > as it
> > > > > > > > > > > > is a
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right away
> > as
> > > > > there
> > > > > > is
> > > > > > > > no
> > > > > > > > > > > change
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10
> is
> > > > going
> > > > > to
> > > > > > > be
> > > > > > > > > code
> > > > > > > > > > > > > freeze
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > late Nov and let's say we give a month to the
> > > > > > development
> > > > > > > > and
> > > > > > > > > > > > review
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27 by
> > late
> > > > > Oct.
> > > > > > > > There
> > > > > > > > > are
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > > weeks. Personally I think it is doable. If
> > > FLIP-27
> > > > is
> > > > > > not
> > > > > > > > > ready
> > > > > > > > > > > by
> > > > > > > > > > > > > late
> > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar
> > connector
> > > > with
> > > > > > the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > interface. This means we will have Pulsar
> > > connector
> > > > > in
> > > > > > > > Flink
> > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > either
> > > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Because we are going to have Pulsar sink and
> > > source
> > > > > > > checked
> > > > > > > > > in
> > > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > > it might make sense to have two FLIPs, one
> for
> > > > Pulsar
> > > > > > > sink
> > > > > > > > > and
> > > > > > > > > > > > > another
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Pulsar source. And we can start the work on
> > > Pulsar
> > > > > sink
> > > > > > > > right
> > > > > > > > > > > away.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > What's the take from Flink community? Shall
> > we
> > > > wait
> > > > > > for
> > > > > > > > > FLIP-27
> > > > > > > > > > > > or
> > > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > > proceed to next steps? And what the next
> > steps
> > > > are?
> > > > > > :-)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think having a Pulsar connector in
> Flink
> > > can
> > > > > be a
> > > > > > > > good
> > > > > > > > > > > mutual
> > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Another perspective is that Pulsar
> > connector
> > > is
> > > > > the
> > > > > > > 1st
> > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > > > management
> > > > > > > system
> > > > > > > > > and
> > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > > It'll be cool to see how the integration
> > > turns
> > > > > out
> > > > > > > and
> > > > > > > > > > > whether
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> > > > currently
> > > > > in
> > > > > > > > > Beta, to
> > > > > > > > > > > > > cater
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in favor
> of
> > > > > merging
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized PRs,
> > e.g.
> > > > > maybe
> > > > > > > one
> > > > > > > > > for
> > > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > > source/sink functionalities and another
> for
> > > > > schema
> > > > > > > and
> > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for
> > FLIP-27.
> > > > But
> > > > > I
> > > > > > > > don't
> > > > > > > > > > > think
> > > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > > should be a blocker in cases where it
> > cannot
> > > > make
> > > > > > its
> > > > > > > > way
> > > > > > > > > > > into
> > > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time
> for
> > > > > > > committers
> > > > > > > > to
> > > > > > > > > > > > review
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > > > > interfaces.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket
> Qin <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > You are right. It all depends on when
> the
> > > new
> > > > > > > source
> > > > > > > > > > > > interface
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > be ready. Personally I think it would
> be
> > > > there
> > > > > in
> > > > > > > > > about a
> > > > > > > > > > > > month
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > > I could be too optimistic. It would
> also
> > be
> > > > > good
> > > > > > to
> > > > > > > > > hear
> > > > > > > > > > > what
> > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > > and Stephan think as they are also
> > involved
> > > > in
> > > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > In general I think we should have
> Pulsar
> > > > > > connector
> > > > > > > in
> > > > > > > > > Flink
> > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > > preferably with the new source
> interface.
> > > We
> > > > > can
> > > > > > > also
> > > > > > > > > check
> > > > > > > > > > > > it
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > > with old source interface, but I
> suspect
> > > few
> > > > > > users
> > > > > > > > > will use
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > next official release. Therefore, it
> > seems
> > > > > > > reasonable
> > > > > > > > > to
> > > > > > > > > > > > wait a
> > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > to see whether we can jump to the new
> > > source
> > > > > > > > > interface. As
> > > > > > > > > > > > long
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a
> little
> > > bit
> > > > > > > doesn't
> > > > > > > > > seem
> > > > > > > > > > > to
> > > > > > > > > > > > > hurt
> > > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till
> > > Rohrmann
> > > > <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'm wondering what the problem would
> be
> > > if
> > > > we
> > > > > > > > > committed
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > connector before the new source
> > interface
> > > > is
> > > > > > > ready.
> > > > > > > > > If I
> > > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > correctly, then we need to support
> the
> > > old
> > > > > > source
> > > > > > > > > > > interface
> > > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > existing connectors. By checking it
> in
> > > > early
> > > > > I
> > > > > > > > could
> > > > > > > > > see
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > our users could start using the
> > connector
> > > > > > > earlier.
> > > > > > > > > > > > Moreover,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration
> is
> > > > being
> > > > > > > > delayed
> > > > > > > > > in
> > > > > > > > > > > > case
> > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > source interface should be delayed.
> The
> > > > only
> > > > > > > > > downside I
> > > > > > > > > > > see
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > > review effort and potential fixes
> which
> > > > might
> > > > > > be
> > > > > > > > > > > irrelevant
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > source interface implementation. I
> > guess
> > > it
> > > > > > > mainly
> > > > > > > > > > > depends
> > > > > > > > > > > > on
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > > we are when the new source interface
> > will
> > > > be
> > > > > > > ready.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM Becket
> > > Qin <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Just want to have some update on
> > > FLIP-27.
> > > > > > > > Although
> > > > > > > > > the
> > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > discussion thread has been quiet
> for
> > > some
> > > > > > > time, a
> > > > > > > > > few
> > > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > > contributors in Flink community
> were
> > > > > actually
> > > > > > > > > > > prototyping
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > > We have made some good progress
> there
> > > but
> > > > > > want
> > > > > > > to
> > > > > > > > > > > update
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > > after the entire thing is verified
> to
> > > > work
> > > > > in
> > > > > > > > case
> > > > > > > > > > > there
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > > minute surprise in the
> > implementation.
> > > I
> > > > > > don't
> > > > > > > > > have an
> > > > > > > > > > > > > exact
> > > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > > guess it is going to be within a
> > month
> > > or
> > > > > so.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I am happy to review the current
> > Flink
> > > > > Pulsar
> > > > > > > > > connector
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good to
> > > avoid
> > > > > the
> > > > > > > > case
> > > > > > > > > that
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Pulsar connector with some review
> > > efforts
> > > > > and
> > > > > > > > > shortly
> > > > > > > > > > > > after
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM
> Yijie
> > > > Shen <
> > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> > > > connector
> > > > > > has
> > > > > > > > > always
> > > > > > > > > > > > been
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > users with the latest features of
> > > both
> > > > > > > systems
> > > > > > > > as
> > > > > > > > > > > soon
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > propose to contribute the
> connector
> > > to
> > > > > > Flink
> > > > > > > > and
> > > > > > > > > hope
> > > > > > > > > > > > to
> > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from
> Flink
> > > > > experts
> > > > > > > to
> > > > > > > > > ensure
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its
> > existence
> > > > at
> > > > > > the
> > > > > > > > > > > beginning
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > > the connector implementation
> based
> > on
> > > > > Flink
> > > > > > > > 1.9;
> > > > > > > > > we
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > build a connector that supports
> > both
> > > > > batch
> > > > > > > and
> > > > > > > > > stream
> > > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > > However, it has been inactive for
> > > some
> > > > > > time,
> > > > > > > so
> > > > > > > > > we
> > > > > > > > > > > > > decided
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > a connector with most of the new
> > > > > features,
> > > > > > > such
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > > and the new catalog API first. We
> > > will
> > > > > pay
> > > > > > > > > attention
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> > > incorporate
> > > > it
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Regarding the test status of the
> > > > > connector,
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> > repository
> > > > and
> > > > > > > aimed
> > > > > > > > to
> > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > > tests as we could. We are also
> > happy
> > > to
> > > > > > hear
> > > > > > > > > > > > suggestions
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > supervision from the Flink
> > community
> > > to
> > > > > > > improve
> > > > > > > > > the
> > > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > > performance of the connector
> > > > > continuously.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM
> > Sijie
> > > > Guo
> > > > > <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the
> comments
> > > and
> > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > It seems to me that the main
> > > question
> > > > > > here
> > > > > > > is
> > > > > > > > > > > about -
> > > > > > > > > > > > > > "how
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > community maintain the
> > connector?".
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from
> > myself.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to
> host
> > > this
> > > > > > > > > integration
> > > > > > > > > > > is
> > > > > > > > > > > > > kind
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > > here. I believe there can be
> many
> > > > ways
> > > > > to
> > > > > > > > > achieve
> > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > As part of the contribution,
> what
> > > we
> > > > > are
> > > > > > > > > looking
> > > > > > > > > > > for
> > > > > > > > > > > > > here
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > > communities can build the
> > > > collaboration
> > > > > > > > > > > relationship
> > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > > the integration between Pulsar
> > and
> > > > > Flink.
> > > > > > > > Even
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > try
> > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > > > community.
> > > > > We
> > > > > > > are
> > > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > > facing the fact that we have
> less
> > > > > > > experiences
> > > > > > > > > in
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > community. In order to make
> sure
> > we
> > > > > > > maintain
> > > > > > > > > and
> > > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > > > integration
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > users
> > > > > > > > > > > > > who
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > technologies, we need some help
> > > from
> > > > > the
> > > > > > > > > experts
> > > > > > > > > > > from
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > 2) We have been following
> FLIP-27
> > > > for a
> > > > > > > > while.
> > > > > > > > > > > > > Originally
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > > of contributing the connectors
> > back
> > > > > after
> > > > > > > > > > > integrating
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > new API introduced in FLIP-27.
> > But
> > > we
> > > > > > > decided
> > > > > > > > > to
> > > > > > > > > > > > > initiate
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > > as early as possible. Because
> we
> > > > > believe
> > > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > > it now rather than later. As
> part
> > > of
> > > > > > > > > contribution,
> > > > > > > > > > > it
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > community understand more about
> > > > Pulsar
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink
> > > community
> > > > > > > verify
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g. catalog
> > > API).
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24 AM
> > > Becket
> > > > > > Qin <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > > > > contributing
> > > > > > > the
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > In general, I think having
> > Pulsar
> > > > > > > connector
> > > > > > > > > with
> > > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink.
> So
> > I
> > > am
> > > > > > happy
> > > > > > > > the
> > > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also like
> to
> > > > > provide
> > > > > > > > some
> > > > > > > > > > > > context
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors
> ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > The current way Flink
> maintains
> > > its
> > > > > > > > > connector has
> > > > > > > > > > > > hit
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > > With more and more connectors
> > > > coming
> > > > > > into
> > > > > > > > > Flink
> > > > > > > > > > > > repo,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > > problems such as long build
> and
> > > > > testing
> > > > > > > > > time. To
> > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the
> > > following:
> > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors
> > into
> > > a
> > > > > > > separate
> > > > > > > > > > > > > repository.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential
> > solution
> > > > to
> > > > > > > > shorten
> > > > > > > > > the
> > > > > > > > > > > > > build
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors
> to
> > > stay
> > > > > as
> > > > > > > > > ecosystem
> > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > > > > functionality
> > > > > > > > and
> > > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > > has driven to create a Flink
> > > > > Ecosystem
> > > > > > > > > project
> > > > > > > > > > > > > website
> > > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > > through some final approval
> > > > process.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it
> > would
> > > > be
> > > > > > > great
> > > > > > > > to
> > > > > > > > > > > first
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an
> > ecosystem
> > > > > > project
> > > > > > > > with
> > > > > > > > > > > great
> > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink
> > Pulsar
> > > > > > > connector
> > > > > > > > > is
> > > > > > > > > > > > tested
> > > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > we can learn something to
> > > maintain
> > > > it
> > > > > > as
> > > > > > > an
> > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > > quality and test coverage. If
> > the
> > > > > > quality
> > > > > > > > as
> > > > > > > > > an
> > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as well
> > > adopt
> > > > it
> > > > > > > into
> > > > > > > > > the
> > > > > > > > > > > main
> > > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort
> is
> > > > > FLIP-27
> > > > > > > > where
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > > architecture
> > > > > and
> > > > > > > > > > > interface.
> > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore
> timing
> > > > wise,
> > > > > if
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > going
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I am
> > > > > wondering
> > > > > > if
> > > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector
> adapt
> > to
> > > > the
> > > > > > new
> > > > > > > > > > > interface
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32
> PM
> > > > > Chesnay
> > > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we
> may
> > > end
> > > > > up
> > > > > > > > > repeating
> > > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > There were already 2
> attempts
> > > at
> > > > > > > > > contributing a
> > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > of which failed because no
> > > > > committer
> > > > > > > was
> > > > > > > > > > > getting
> > > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a
> > dedicated
> > > > > > > > discussion
> > > > > > > > > > > thread
> > > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting
> > several
> > > > +1's
> > > > > > > from
> > > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > We should really make sure
> > that
> > > > if
> > > > > we
> > > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > contribution it will
> actually
> > > get
> > > > > the
> > > > > > > > > attention
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> > > > recommend
> > > > > > > > > maintaining
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to it
> > from
> > > > the
> > > > > > > > > > > documentation
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page for
> > > > sharing
> > > > > > > > > artifacts
> > > > > > > > > > > > among
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > > the state of that anyway?),
> > > this
> > > > > may
> > > > > > > be a
> > > > > > > > > > > better
> > > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> > > > Rohrmann
> > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for starting
> > > this
> > > > > > > > discussion
> > > > > > > > > > > > Yijie.
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a very
> > > > > valuable
> > > > > > > > > addition
> > > > > > > > > > > > since
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it would
> > > > further
> > > > > > > > expand
> > > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > > from a project
> perspective
> > it
> > > > > makes
> > > > > > > > > sense for
> > > > > > > > > > > > me
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > My main concern/question
> is
> > > how
> > > > > can
> > > > > > > the
> > > > > > > > > Flink
> > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen
> in
> > > the
> > > > > past
> > > > > > > > that
> > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > > actively developed
> > components
> > > > > > because
> > > > > > > > > they
> > > > > > > > > > > need
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > external system and with
> > > Flink.
> > > > > > Given
> > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > > to help with maintaining,
> > > > > improving
> > > > > > > and
> > > > > > > > > > > > evolving
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can
> > > achieve
> > > > > > this.
> > > > > > > > > Hence,
> > > > > > > > > > > +1
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> 2:03
> > > AM
> > > > > > Sijie
> > > > > > > > Guo
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main
> > driver
> > > > > behind
> > > > > > > > > > > FLINK-9641
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on
> this.
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and
> FLINK-9168
> > > was
> > > > > > > created
> > > > > > > > > for
> > > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > > > integration
> > > > > > was
> > > > > > > > done
> > > > > > > > > > > with
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year
> ago
> > > and
> > > > we
> > > > > > > ended
> > > > > > > > > up
> > > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar users
> to
> > > use
> > > > > > Flink
> > > > > > > > to
> > > > > > > > > > > > process
> > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > >
> > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is
> pretty
> > > > simple
> > > > > > and
> > > > > > > > > there
> > > > > > > > > > > is
> > > > > > > > > > > > no
> > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we
> have
> > > > made a
> > > > > > lot
> > > > > > > > of
> > > > > > > > > > > > changes
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > > > first-class
> > > > > > > > > citizen in
> > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > >> other computing engines
> > for
> > > > > > > processing
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink how
> > to
> > > > > > > integrate
> > > > > > > > > with
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> > pulsar-flink
> > > > > > > > connectors
> > > > > > > > > from
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and
> > catalog
> > > > API
> > > > > as
> > > > > > > the
> > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that
> > being
> > > > > said,
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > new
> > > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > > >> you can register pulsar
> > as a
> > > > > flink
> > > > > > > > > catalog
> > > > > > > > > > > and
> > > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink SQL.
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example about
> > how
> > > > to
> > > > > > use
> > > > > > > > > Pulsar
> > > > > > > > > > > as
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also written a
> > > blog
> > > > > post
> > > > > > > > > > > explaining
> > > > > > > > > > > > > why
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink 1.9
> > and
> > > > > what
> > > > > > > are
> > > > > > > > > the
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is not
> > > just
> > > > a
> > > > > > > simple
> > > > > > > > > data
> > > > > > > > > > > > sink
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a fully
> > > > > integrated
> > > > > > > > > streaming
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > > > schema/catalog
> > > > > > > > and
> > > > > > > > > > > > state).
> > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create a
> > > great
> > > > > > > > streaming
> > > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first, unified
> > > data
> > > > > > > > > processing.
> > > > > > > > > > > > Since
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> > > integration
> > > > to
> > > > > > > Flink
> > > > > > > > > here,
> > > > > > > > > > > > we
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and
> > evolve
> > > > the
> > > > > > > > > integration
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and
> > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a bit
> > > more
> > > > > > > > background
> > > > > > > > > > > about
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me know
> > > what
> > > > > are
> > > > > > > your
> > > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at
> > 11:54
> > > > AM
> > > > > > Yun
> > > > > > > > > Tang <
> > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar
> > > becomes
> > > > > > more
> > > > > > > > and
> > > > > > > > > more
> > > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people
> > willing
> > > to
> > > > > > > > > contribute to
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > > > discussion,
> > > > > > > would
> > > > > > > > > you
> > > > > > > > > > > > please
> > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship
> between
> > > this
> > > > > > > thread
> > > > > > > > to
> > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink [2]
> > > > > > connector?
> > > > > > > > > Will the
> > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> > > > different
> > > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > >
> https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday,
> September
> > 3,
> > > > > 2019
> > > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> [hidden email]
> > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS]
> > > Contribute
> > > > > > > Pulsar
> > > > > > > > > Flink
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open
> the
> > > > > > discussion
> > > > > > > > of
> > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to
> > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief introduction
> > to
> > > > > Apache
> > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > > > > multi-tenant,
> > > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging
> system.
> > > > > Pulsar
> > > > > > > > > includes
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for
> > multiple
> > > > > > > clusters
> > > > > > > > > in a
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> geo-replication
> > of
> > > > > > > messages
> > > > > > > > > across
> > > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end latency,
> > > > > seamless
> > > > > > > > > > > scalability
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed message
> > > > delivery
> > > > > > > with
> > > > > > > > > > > > persistent
> > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > > > BookKeeper.
> > > > > > > > > Nowadays,
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> > companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of Pulsar
> > > Flink
> > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink
> > connector
> > > we
> > > > > are
> > > > > > > > > planning
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and Pulsar
> > > 2.4.0.
> > > > > The
> > > > > > > > main
> > > > > > > > > > > > features
> > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a streaming
> > > > source
> > > > > > with
> > > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming
> results
> > to
> > > > > > Pulsar
> > > > > > > > with
> > > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > > > > exactly-once
> > > > > > > as
> > > > > > > > > well
> > > > > > > > > > > > when
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features
> > ready
> > > in
> > > > > its
> > > > > > > > 2.5.0
> > > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink new
> > > Table
> > > > > API
> > > > > > > Type
> > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> > (de)serialize
> > > > > > > messages
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with Flink
> > new
> > > > > > Catalog
> > > > > > > > API
> > > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics as
> > > > tables
> > > > > in
> > > > > > > > > Table API
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Stephan Ewen]

My assumption is as Sijie's, that once the connector is either part of
Flink, or part of the streamnative repo. No double maintenance.

I feel this discussion is very much caught in problems that are all
solvable if we want to solve them.
Maybe we can think what our goal for users and the communities is?

  - Do we want to help build a relationship between the Pulsar and Flink
open source communities?
  - Will users find a connector in the streamnative repository?
  - Will users trust a connector that is not part of Flink as much?

And then decide what is best according to the overall goals there.
As mentioned before, I believe that we can handle connectors more
pragmatically and less strict than the core of Flink, if it helps unlocking
users.

Best,
Stephan



On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]> wrote:

> Thanks Becket.
>
> I think it is better for the Flink community to judge the benefits of doing
> this. I was trying to provide some views from outsiders.
>
> Thanks,
> Sijie
>
> On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]> wrote:
>
> > Hi Sijie,
> >
> > Yes, we will have to support existing old connectors and new connectors
> in
> > parallel for a while. We have to take that maintenance overhead because
> > existing connectors have been used by the users for a long time. I guess
> It
> > may take at least a year for us to fully remove the old connectors.
> >
> > Process wise, we can do the same for Pulsar connector. But I am not sure
> if
> > we want to have the same burden on Pulsar connector, and I would like to
> > understand the benefit of doing that.
> >
> > For users, the benefit of having the old Pulsar connector checked in
> seems
> > limited because 1) that code base will be immediately deprecated in the
> > next release in 3-4 months; 2) users can always use it even if it is not
> in
> > the Flink code base. Admittedly it is not as convenient as having it in
> > Flink code base, but doesn't seem super either. And after 3-4 months,
> users
> > can just use the new connector in Flink repo.
> >
> > For Flink developers, the old connector code base is not something that
> we
> > want to evolve later. Instead, these code will be deprecated and
> > removed. So why do we want to get a beta version out to attract people to
> > use something we don't want to maintain?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]> wrote:
> >
> > > Thanks everyone here. Sorry for jumping into the discussion here.
> > >
> > > I am not very familiar about the deprecation process in Flink. If I
> > > misunderstood the process, please fix me.
> > >
> > > As far as I understand, FLIP-27 is introducing a new unified API for
> > > connectors. After it introduces the new API
> > > and before moving all the existing connectors from old API to new API,
> > both
> > > old ApI and new API will co-exist
> > > for a while until Flink moves all existing connectors to new API. So
> the
> > > Pulsar connector (using old API) can
> > > follow the deprecation process with other connector using old API and
> the
> > > deprecation of old API, no?
> > >
> > > If that's the case, I think contributing the current connector back to
> > > Flink rather than maintaining it outside Flink
> > > would provide a bit more benefits. We can deprecate the existing
> > > streamnative/pulsar-flink repo and point the users
> > > to use the connector in Flink repo. So all the review processes will
> > happen
> > > within Flink for both old connector and
> > > new connector. It also reduces the confusions for the users as the
> > > documentation and code base happen in one place.
> > >
> > > Thoughts?
> > >
> > > - Sijie
> > >
> > >
> > >
> > >
> > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <[hidden email]>
> > wrote:
> > >
> > > > Thanks for the explanation, Stephan. I have a few questions /
> thoughts.
> > > >
> > > > So that means we will remove the old connector without a major
> version
> > > > bump, is that correct?
> > > >
> > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors
> will
> > > > always work because we saw some dependency class collisions in the
> > past.
> > > To
> > > > make it safe we may have to maintain the old code for one more
> release.
> > > >
> > > > To be honest I am still wondering if we have to put the old connector
> > in
> > > > Flink repo. if we check in the old connector to Flink. We will end up
> > in
> > > > the following situation:
> > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > 2. Old connector in Flink Repo, which may be different from the one
> in
> > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> > > > 3. New connector in Flink Repo.
> > > >
> > > > We need to think about how to make the users in each case happy.
> > > > - For users of (1), I assume Sijie and Yijie will have to maintain
> the
> > > code
> > > > a bit longer for its own compatibility even after we have (2). In
> that
> > > > case, bugs found in old connector may or may not need to be fixed in
> > both
> > > > Flink and the streamnative/pulsar-flink repo.
> > > > - For users of (2), will we provide bug fixes? If we do, it will be a
> > > > little awkward because those bug fixes will be immediately deprecated
> > in
> > > > 1.11, and removed in 1.12. So we are essentially asking users to
> > migrate
> > > > away from the bug fix. After Flink 1.12, users may still have to
> switch
> > > to
> > > > use (3) due to the potential dependency class conflicts mentioned
> > above.
> > > > - Users of (3) have a much easier life and don't need to worry too
> > much.
> > > >
> > > > The above story seems a little complicated to tell. I think it will
> be
> > > much
> > > > easier to not have (2) at all.
> > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > 3. New connector in Flink Repo.
> > > >
> > > > - Old connector will only be maintained in streamnative/pulsar-flink
> > repo
> > > > until it is fully deprecated. Users can always use the existing
> Pulsar
> > > > connector in that repo.
> > > > - New connector will be in Flink repo and maintained like the other
> > > > connectors.
> > > >
> > > > This seems much simpler for users understand and they will not be
> > blocked
> > > > from using the old connector. If the concern is about the quality of
> > the
> > > > connector in streamnative/pulsar-flink repo, is it enough for us just
> > to
> > > > review the code in streamnative/pulsar-flink connector to make sure
> it
> > > > looks good from Flink's perspective?
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]>
> wrote:
> > > >
> > > > > My take would be the following:
> > > > >
> > > > >   - If we merge the connector now and replace it with a FLIP-27
> > version
> > > > > before the 1.10 release, then we need no deprecation process
> > > > >   - If we don't manage to replace it with a FLIP-27 version before
> > the
> > > > 1.10
> > > > > release, than it is good that we have the other version, so no
> users
> > > get
> > > > > blocked.
> > > > >
> > > > > In the latter case we can see how we want to do it. Immediate
> removal
> > > of
> > > > > the old version or deprecation label and keeping it for one more
> > > release.
> > > > > Given that you should be able to use a Flink 1.10 connector with
> > Flink
> > > > 1.11
> > > > > as well (stable public APIs) there is also a workaround if you need
> > an
> > > > old
> > > > > connector in a newer version. So immediate removal might even be
> > > > feasible.
> > > > >
> > > > >
> > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Hi Stephan,
> > > > > >
> > > > > > Thanks for the clarification. I completely agree with you and
> > Thomas
> > > on
> > > > > the
> > > > > > process of adding connectors to Flink repo. However, I am
> wondering
> > > > what
> > > > > is
> > > > > > the deprecation process? Given the main concern here was that we
> > may
> > > > have
> > > > > > to maintain two Pulsar connector code bases until the old one is
> > > > removed
> > > > > > from the repo, it would be good to know how long we have to do
> > that.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > > Some quick thoughts on the connector contribution process. I
> > > > basically
> > > > > > > reiterate here what Thomas mentioned in another thread about
> the
> > > > > Kinesis
> > > > > > > connector.
> > > > > > >
> > > > > > > For connectors, we should favor a low-overhead contribution
> > > process,
> > > > > and
> > > > > > > accept user code and changes more readily than in the core
> > system.
> > > > > > > That is because connectors have both a big variety of scenarios
> > > they
> > > > > get
> > > > > > > used in (only through use and many small contributions do they
> > > become
> > > > > > > really useful over time) and at the same time, and committers
> do
> > > not
> > > > > use
> > > > > > > the connector themselves and usually cannot foresee too well
> what
> > > is
> > > > > > > needed.
> > > > > > >
> > > > > > > Further more, a missing connector (or connector feature) is
> > often a
> > > > > > bigger
> > > > > > > show stopper for users than a missing API or system feature.
> > > > > > >
> > > > > > > Along these lines of thougt, the conclusion would be to take
> the
> > > > Pulsar
> > > > > > > connector now, focus the review on legal/dependencies/rough
> code
> > > > style
> > > > > > and
> > > > > > > conventions, label it as "beta" (in the sense of "new code"
> that
> > is
> > > > > "not
> > > > > > > yet tested through longer use") and go ahead. And then evolve
> it
> > > > > quickly
> > > > > > > without putting formal blockers in the way, meaning also
> adding a
> > > new
> > > > > > FLIP
> > > > > > > 27 version when it is there.
> > > > > > >
> > > > > > > Best,
> > > > > > > Stephan
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yijie,
> > > > > > > >
> > > > > > > > Could you please follow the FLIP process to start a new FLIP
> > > > > > [DISCUSSION]
> > > > > > > > thread in the mailing list?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > > > >
> > > > > > > > I see two FLIP-69 discussion in the mailing list now. So
> there
> > > is a
> > > > > > FLIP
> > > > > > > > number collision. Can you change the FLIP number to 72?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yijie,
> > > > > > > > >
> > > > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > > > Would you mind enabling comments/suggestions on the google
> > doc
> > > > > link?
> > > > > > > This
> > > > > > > > > way the contributors from the community can comment on the
> > doc.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Rong
> > > > > > > > >
> > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > I've drafted a FLIP that describes the current design of
> > the
> > > > > Pulsar
> > > > > > > > > > connector:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > > > >
> > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Yijie
> > > > > > > > > >
> > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > Sorry for joining the discussion late and thanks Yijie
> &
> > > > Sijie
> > > > > > for
> > > > > > > > > > driving
> > > > > > > > > > > the discussion.
> > > > > > > > > > > I also think the Pulsar connector would be a very
> > valuable
> > > > > > addition
> > > > > > > > to
> > > > > > > > > > > Flink. I can also help out a bit on the review side :-)
> > > > > > > > > > >
> > > > > > > > > > > Regarding the timeline, I also share concerns with
> Becket
> > > on
> > > > > the
> > > > > > > > > > > relationship between the new Pulsar connector and
> > FLIP-27.
> > > > > > > > > > > There's also another discussion just started by Stephan
> > on
> > > > > > dropping
> > > > > > > > > Kafka
> > > > > > > > > > > 9/10 support on next Flink release [1].  Although the
> > > > situation
> > > > > > is
> > > > > > > > > > somewhat
> > > > > > > > > > > different, and Kafka 9/10 connector has been in Flink
> for
> > > > > almost
> > > > > > > 3-4
> > > > > > > > > > years,
> > > > > > > > > > > based on the discussion I am not sure if a major
> version
> > > > > release
> > > > > > > is a
> > > > > > > > > > > requirement for removing old connector supports.
> > > > > > > > > > >
> > > > > > > > > > > I think there shouldn't be a blocker if we agree the
> old
> > > > > > connector
> > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > removed once FLIP-27 based Pulsar connector is there.
> As
> > > > > Stephan
> > > > > > > > > stated,
> > > > > > > > > > it
> > > > > > > > > > > is easier to contribute the source sooner and adjust it
> > > > later.
> > > > > > > > > > > We should also ensure we clearly communicate the
> message:
> > > for
> > > > > > > > example,
> > > > > > > > > > > putting an experimental flag on the pre-FLIP27
> connector
> > > page
> > > > > of
> > > > > > > the
> > > > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Rong
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Technically speaking, removing the old connector code
> > is
> > > a
> > > > > > > > backwards
> > > > > > > > > > > > incompatible change which requires a major version
> > bump,
> > > > i.e.
> > > > > > > Flink
> > > > > > > > > > 2.x.
> > > > > > > > > > > > Given that we don't have a clear plan on when to have
> > the
> > > > > next
> > > > > > > > major
> > > > > > > > > > > > version release, it seems unclear how long the old
> > > > connector
> > > > > > code
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > there if we check it in right now. Or will we remove
> > the
> > > > old
> > > > > > > > > connector
> > > > > > > > > > > > without a major version bump? In any case, it sounds
> > not
> > > > > quite
> > > > > > > user
> > > > > > > > > > > > friendly to the those who might use the old Pulsar
> > > > > connector. I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > > sure
> > > > > > > > > > > > if it is worth these potential problems in order to
> > have
> > > > the
> > > > > > > Pulsar
> > > > > > > > > > source
> > > > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Agreed, if we check in the old code, we should make
> > it
> > > > > clear
> > > > > > > that
> > > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > > > > be removed as soon as the FLIP-27 based version of
> > the
> > > > > > > connector
> > > > > > > > is
> > > > > > > > > > > > there.
> > > > > > > > > > > > > We should not commit to maintaining the old
> version,
> > > that
> > > > > > would
> > > > > > > > be
> > > > > > > > > > indeed
> > > > > > > > > > > > > too much overhead.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yes, the overhead would just be review capacity.
> In
> > > > > fact, I
> > > > > > > am
> > > > > > > > > not
> > > > > > > > > > > > > worrying
> > > > > > > > > > > > > > too much about the review capacity. That is just
> a
> > > one
> > > > > time
> > > > > > > > cost.
> > > > > > > > > > My
> > > > > > > > > > > > > > concern is mainly about the long term burden.
> > Assume
> > > we
> > > > > > have
> > > > > > > > new
> > > > > > > > > > source
> > > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > > > > connectors
> > > > > > in
> > > > > > > > old
> > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new
> > > source
> > > > > > > > interface,
> > > > > > > > > > the
> > > > > > > > > > > > old
> > > > > > > > > > > > > > Pulsar connector might be deprecated almost
> > > immediately
> > > > > > after
> > > > > > > > > > checked
> > > > > > > > > > > > in,
> > > > > > > > > > > > > > but we may still have to maintain two code bases.
> > For
> > > > the
> > > > > > > > > existing
> > > > > > > > > > > > > > connectors, we have to do that anyways. But it
> > would
> > > be
> > > > > > good
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > > > > > introducing a new connector with the same
> problem.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Nice to see this lively discussion about the
> > Pulsar
> > > > > > > > connector.
> > > > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a
> community
> > > > > package
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Looks like the discussion is more going towards
> > > > > > > > contribution. I
> > > > > > > > > > think
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > is good, especially if we think that we want to
> > > > build a
> > > > > > > > > similarly
> > > > > > > > > > > > deep
> > > > > > > > > > > > > > > integration with Pulsar as we have for example
> > with
> > > > > > Kafka.
> > > > > > > > The
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > already looks like a more thorough connector
> than
> > > > many
> > > > > > > others
> > > > > > > > > we
> > > > > > > > > > have
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the repository.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > With either a repo split, or the new build
> > system,
> > > I
> > > > > hope
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > > build
> > > > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Becket offered some help already, I can also
> > help a
> > > > > bit.
> > > > > > I
> > > > > > > > hope
> > > > > > > > > > that
> > > > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making some
> > > > PoC-ing
> > > > > > > > > > progress, but
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > take 2 more months, I would estimate, before it
> > is
> > > > > fully
> > > > > > > > > > available.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If we want to be on the safe side with the
> > > > > contribution,
> > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > > > > contribute the source sooner and adjust it
> later.
> > > > That
> > > > > > > would
> > > > > > > > > also
> > > > > > > > > > > > help
> > > > > > > > > > > > > us
> > > > > > > > > > > > > > > in case things get crazy towards the 1.10
> feature
> > > > > freeze
> > > > > > > and
> > > > > > > > it
> > > > > > > > > > would
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > > > > What would be the overhead of contributing now?
> > > Given
> > > > > > that
> > > > > > > > the
> > > > > > > > > > code
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > already there, it looks like it would be only
> > > review
> > > > > > > > capacity,
> > > > > > > > > > right?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your attention and the promotion
> of
> > > this
> > > > > > work.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible
> for
> > > more
> > > > > > > > specific
> > > > > > > > > > > > > > discussions.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not
> reached
> > a
> > > > > > > consensus.
> > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > I will explain all the functionalities of the
> > > > > existing
> > > > > > > > > > connector in
> > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and
> Catalog)
> > to
> > > > > > > continue
> > > > > > > > > our
> > > > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If we agree that the goal is to have Pulsar
> > > > > connector
> > > > > > > in
> > > > > > > > > > 1.10,
> > > > > > > > > > > > how
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector to
> > > Flink
> > > > > main
> > > > > > > > repo
> > > > > > > > > > as it
> > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right
> away
> > > as
> > > > > > there
> > > > > > > is
> > > > > > > > > no
> > > > > > > > > > > > change
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink 1.10
> > is
> > > > > going
> > > > > > to
> > > > > > > > be
> > > > > > > > > > code
> > > > > > > > > > > > > > freeze
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > late Nov and let's say we give a month to
> the
> > > > > > > development
> > > > > > > > > and
> > > > > > > > > > > > > review
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27
> by
> > > late
> > > > > > Oct.
> > > > > > > > > There
> > > > > > > > > > are
> > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > > > weeks. Personally I think it is doable. If
> > > > FLIP-27
> > > > > is
> > > > > > > not
> > > > > > > > > > ready
> > > > > > > > > > > > by
> > > > > > > > > > > > > > late
> > > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar
> > > connector
> > > > > with
> > > > > > > the
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > interface. This means we will have Pulsar
> > > > connector
> > > > > > in
> > > > > > > > > Flink
> > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > either
> > > > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Because we are going to have Pulsar sink
> and
> > > > source
> > > > > > > > checked
> > > > > > > > > > in
> > > > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > > > it might make sense to have two FLIPs, one
> > for
> > > > > Pulsar
> > > > > > > > sink
> > > > > > > > > > and
> > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Pulsar source. And we can start the work on
> > > > Pulsar
> > > > > > sink
> > > > > > > > > right
> > > > > > > > > > > > away.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > What's the take from Flink community?
> Shall
> > > we
> > > > > wait
> > > > > > > for
> > > > > > > > > > FLIP-27
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > > > proceed to next steps? And what the next
> > > steps
> > > > > are?
> > > > > > > :-)
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen Li <
> > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I think having a Pulsar connector in
> > Flink
> > > > can
> > > > > > be a
> > > > > > > > > good
> > > > > > > > > > > > mutual
> > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar
> > > connector
> > > > is
> > > > > > the
> > > > > > > > 1st
> > > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > > > > management
> > > > > > > > system
> > > > > > > > > > and
> > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > > > It'll be cool to see how the
> integration
> > > > turns
> > > > > > out
> > > > > > > > and
> > > > > > > > > > > > whether
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which are
> > > > > currently
> > > > > > in
> > > > > > > > > > Beta, to
> > > > > > > > > > > > > > cater
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in
> favor
> > of
> > > > > > merging
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized
> PRs,
> > > e.g.
> > > > > > maybe
> > > > > > > > one
> > > > > > > > > > for
> > > > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > > > source/sink functionalities and another
> > for
> > > > > > schema
> > > > > > > > and
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for
> > > FLIP-27.
> > > > > But
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > > think
> > > > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > > > should be a blocker in cases where it
> > > cannot
> > > > > make
> > > > > > > its
> > > > > > > > > way
> > > > > > > > > > > > into
> > > > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of time
> > for
> > > > > > > > committers
> > > > > > > > > to
> > > > > > > > > > > > > review
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to new
> > > > > > interfaces.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket
> > Qin <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > You are right. It all depends on when
> > the
> > > > new
> > > > > > > > source
> > > > > > > > > > > > > interface
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > be ready. Personally I think it would
> > be
> > > > > there
> > > > > > in
> > > > > > > > > > about a
> > > > > > > > > > > > > month
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > > > I could be too optimistic. It would
> > also
> > > be
> > > > > > good
> > > > > > > to
> > > > > > > > > > hear
> > > > > > > > > > > > what
> > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > > > and Stephan think as they are also
> > > involved
> > > > > in
> > > > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > In general I think we should have
> > Pulsar
> > > > > > > connector
> > > > > > > > in
> > > > > > > > > > Flink
> > > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > > > preferably with the new source
> > interface.
> > > > We
> > > > > > can
> > > > > > > > also
> > > > > > > > > > check
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > > > with old source interface, but I
> > suspect
> > > > few
> > > > > > > users
> > > > > > > > > > will use
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > next official release. Therefore, it
> > > seems
> > > > > > > > reasonable
> > > > > > > > > > to
> > > > > > > > > > > > > wait a
> > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > to see whether we can jump to the new
> > > > source
> > > > > > > > > > interface. As
> > > > > > > > > > > > > long
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a
> > little
> > > > bit
> > > > > > > > doesn't
> > > > > > > > > > seem
> > > > > > > > > > > > to
> > > > > > > > > > > > > > hurt
> > > > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till
> > > > Rohrmann
> > > > > <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem
> would
> > be
> > > > if
> > > > > we
> > > > > > > > > > committed
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > connector before the new source
> > > interface
> > > > > is
> > > > > > > > ready.
> > > > > > > > > > If I
> > > > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > correctly, then we need to support
> > the
> > > > old
> > > > > > > source
> > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > existing connectors. By checking it
> > in
> > > > > early
> > > > > > I
> > > > > > > > > could
> > > > > > > > > > see
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > our users could start using the
> > > connector
> > > > > > > > earlier.
> > > > > > > > > > > > > Moreover,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar integration
> > is
> > > > > being
> > > > > > > > > delayed
> > > > > > > > > > in
> > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > > source interface should be delayed.
> > The
> > > > > only
> > > > > > > > > > downside I
> > > > > > > > > > > > see
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > > > review effort and potential fixes
> > which
> > > > > might
> > > > > > > be
> > > > > > > > > > > > irrelevant
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > source interface implementation. I
> > > guess
> > > > it
> > > > > > > > mainly
> > > > > > > > > > > > depends
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > > > we are when the new source
> interface
> > > will
> > > > > be
> > > > > > > > ready.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM
> Becket
> > > > Qin <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your thoughts.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Just want to have some update on
> > > > FLIP-27.
> > > > > > > > > Although
> > > > > > > > > > the
> > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > discussion thread has been quiet
> > for
> > > > some
> > > > > > > > time, a
> > > > > > > > > > few
> > > > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > > > contributors in Flink community
> > were
> > > > > > actually
> > > > > > > > > > > > prototyping
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > > > We have made some good progress
> > there
> > > > but
> > > > > > > want
> > > > > > > > to
> > > > > > > > > > > > update
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > > > after the entire thing is
> verified
> > to
> > > > > work
> > > > > > in
> > > > > > > > > case
> > > > > > > > > > > > there
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > > > minute surprise in the
> > > implementation.
> > > > I
> > > > > > > don't
> > > > > > > > > > have an
> > > > > > > > > > > > > > exact
> > > > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > > > guess it is going to be within a
> > > month
> > > > or
> > > > > > so.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I am happy to review the current
> > > Flink
> > > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be good
> to
> > > > avoid
> > > > > > the
> > > > > > > > > case
> > > > > > > > > > that
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some review
> > > > efforts
> > > > > > and
> > > > > > > > > > shortly
> > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM
> > Yijie
> > > > > Shen <
> > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback and
> > > > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of the
> > > > > connector
> > > > > > > has
> > > > > > > > > > always
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > users with the latest features
> of
> > > > both
> > > > > > > > systems
> > > > > > > > > as
> > > > > > > > > > > > soon
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > propose to contribute the
> > connector
> > > > to
> > > > > > > Flink
> > > > > > > > > and
> > > > > > > > > > hope
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from
> > Flink
> > > > > > experts
> > > > > > > > to
> > > > > > > > > > ensure
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its
> > > existence
> > > > > at
> > > > > > > the
> > > > > > > > > > > > beginning
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > > > the connector implementation
> > based
> > > on
> > > > > > Flink
> > > > > > > > > 1.9;
> > > > > > > > > > we
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > build a connector that supports
> > > both
> > > > > > batch
> > > > > > > > and
> > > > > > > > > > stream
> > > > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > > > However, it has been inactive
> for
> > > > some
> > > > > > > time,
> > > > > > > > so
> > > > > > > > > > we
> > > > > > > > > > > > > > decided
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > a connector with most of the
> new
> > > > > > features,
> > > > > > > > such
> > > > > > > > > > as
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > > > and the new catalog API first.
> We
> > > > will
> > > > > > pay
> > > > > > > > > > attention
> > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> > > > incorporate
> > > > > it
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Regarding the test status of
> the
> > > > > > connector,
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> > > repository
> > > > > and
> > > > > > > > aimed
> > > > > > > > > to
> > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are also
> > > happy
> > > > to
> > > > > > > hear
> > > > > > > > > > > > > suggestions
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > supervision from the Flink
> > > community
> > > > to
> > > > > > > > improve
> > > > > > > > > > the
> > > > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > > > performance of the connector
> > > > > > continuously.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59 AM
> > > Sijie
> > > > > Guo
> > > > > > <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the
> > comments
> > > > and
> > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the main
> > > > question
> > > > > > > here
> > > > > > > > is
> > > > > > > > > > > > about -
> > > > > > > > > > > > > > > "how
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > community maintain the
> > > connector?".
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from
> > > myself.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to
> > host
> > > > this
> > > > > > > > > > integration
> > > > > > > > > > > > is
> > > > > > > > > > > > > > kind
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > > > here. I believe there can be
> > many
> > > > > ways
> > > > > > to
> > > > > > > > > > achieve
> > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > As part of the contribution,
> > what
> > > > we
> > > > > > are
> > > > > > > > > > looking
> > > > > > > > > > > > for
> > > > > > > > > > > > > > here
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > > > communities can build the
> > > > > collaboration
> > > > > > > > > > > > relationship
> > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > > > the integration between
> Pulsar
> > > and
> > > > > > Flink.
> > > > > > > > > Even
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > > > > community.
> > > > > > We
> > > > > > > > are
> > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we have
> > less
> > > > > > > > experiences
> > > > > > > > > > in
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > community. In order to make
> > sure
> > > we
> > > > > > > > maintain
> > > > > > > > > > and
> > > > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > > > > integration
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > users
> > > > > > > > > > > > > > who
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > technologies, we need some
> help
> > > > from
> > > > > > the
> > > > > > > > > > experts
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following
> > FLIP-27
> > > > > for a
> > > > > > > > > while.
> > > > > > > > > > > > > > Originally
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > > > of contributing the
> connectors
> > > back
> > > > > > after
> > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > new API introduced in
> FLIP-27.
> > > But
> > > > we
> > > > > > > > decided
> > > > > > > > > > to
> > > > > > > > > > > > > > initiate
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > > > as early as possible. Because
> > we
> > > > > > believe
> > > > > > > > > there
> > > > > > > > > > are
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > > > it now rather than later. As
> > part
> > > > of
> > > > > > > > > > contribution,
> > > > > > > > > > > > it
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > community understand more
> about
> > > > > Pulsar
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink
> > > > community
> > > > > > > > verify
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g.
> catalog
> > > > API).
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24
> AM
> > > > Becket
> > > > > > > Qin <
> > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest in
> > > > > > contributing
> > > > > > > > the
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > In general, I think having
> > > Pulsar
> > > > > > > > connector
> > > > > > > > > > with
> > > > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to Flink.
> > So
> > > I
> > > > am
> > > > > > > happy
> > > > > > > > > the
> > > > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also
> like
> > to
> > > > > > provide
> > > > > > > > > some
> > > > > > > > > > > > > context
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors
> > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink
> > maintains
> > > > its
> > > > > > > > > > connector has
> > > > > > > > > > > > > hit
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > > > With more and more
> connectors
> > > > > coming
> > > > > > > into
> > > > > > > > > > Flink
> > > > > > > > > > > > > repo,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > > > problems such as long build
> > and
> > > > > > testing
> > > > > > > > > > time. To
> > > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the
> > > > following:
> > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the connectors
> > > into
> > > > a
> > > > > > > > separate
> > > > > > > > > > > > > > repository.
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential
> > > solution
> > > > > to
> > > > > > > > > shorten
> > > > > > > > > > the
> > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the connectors
> > to
> > > > stay
> > > > > > as
> > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > > > to provide good support for
> > > > > > > functionality
> > > > > > > > > and
> > > > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > > > has driven to create a
> Flink
> > > > > > Ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > > > website
> > > > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > > > through some final approval
> > > > > process.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts, it
> > > would
> > > > > be
> > > > > > > > great
> > > > > > > > > to
> > > > > > > > > > > > first
> > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an
> > > ecosystem
> > > > > > > project
> > > > > > > > > with
> > > > > > > > > > > > great
> > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the Flink
> > > Pulsar
> > > > > > > > connector
> > > > > > > > > > is
> > > > > > > > > > > > > tested
> > > > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > we can learn something to
> > > > maintain
> > > > > it
> > > > > > > as
> > > > > > > > an
> > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > > > quality and test coverage.
> If
> > > the
> > > > > > > quality
> > > > > > > > > as
> > > > > > > > > > an
> > > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as
> well
> > > > adopt
> > > > > it
> > > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > main
> > > > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing effort
> > is
> > > > > > FLIP-27
> > > > > > > > > where
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > > > architecture
> > > > > > and
> > > > > > > > > > > > interface.
> > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore
> > timing
> > > > > wise,
> > > > > > if
> > > > > > > > we
> > > > > > > > > > are
> > > > > > > > > > > > > going
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I
> am
> > > > > > wondering
> > > > > > > if
> > > > > > > > > we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector
> > adapt
> > > to
> > > > > the
> > > > > > > new
> > > > > > > > > > > > interface
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 4:32
> > PM
> > > > > > Chesnay
> > > > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that we
> > may
> > > > end
> > > > > > up
> > > > > > > > > > repeating
> > > > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2
> > attempts
> > > > at
> > > > > > > > > > contributing a
> > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > > of which failed because
> no
> > > > > > committer
> > > > > > > > was
> > > > > > > > > > > > getting
> > > > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a
> > > dedicated
> > > > > > > > > discussion
> > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting
> > > several
> > > > > +1's
> > > > > > > > from
> > > > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > We should really make
> sure
> > > that
> > > > > if
> > > > > > we
> > > > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > contribution it will
> > actually
> > > > get
> > > > > > the
> > > > > > > > > > attention
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined to
> > > > > recommend
> > > > > > > > > > maintaining
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to
> it
> > > from
> > > > > the
> > > > > > > > > > > > documentation
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page
> for
> > > > > sharing
> > > > > > > > > > artifacts
> > > > > > > > > > > > > among
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > > > the state of that
> anyway?),
> > > > this
> > > > > > may
> > > > > > > > be a
> > > > > > > > > > > > better
> > > > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16, Till
> > > > > Rohrmann
> > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for
> starting
> > > > this
> > > > > > > > > discussion
> > > > > > > > > > > > > Yijie.
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a
> very
> > > > > > valuable
> > > > > > > > > > addition
> > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it
> would
> > > > > further
> > > > > > > > > expand
> > > > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > > > from a project
> > perspective
> > > it
> > > > > > makes
> > > > > > > > > > sense for
> > > > > > > > > > > > > me
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream project.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> concern/question
> > is
> > > > how
> > > > > > can
> > > > > > > > the
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have seen
> > in
> > > > the
> > > > > > past
> > > > > > > > > that
> > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > > > actively developed
> > > components
> > > > > > > because
> > > > > > > > > > they
> > > > > > > > > > > > need
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > external system and
> with
> > > > Flink.
> > > > > > > Given
> > > > > > > > > > that
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
> maintaining,
> > > > > > improving
> > > > > > > > and
> > > > > > > > > > > > > evolving
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we can
> > > > achieve
> > > > > > > this.
> > > > > > > > > > Hence,
> > > > > > > > > > > > +1
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> > 2:03
> > > > AM
> > > > > > > Sijie
> > > > > > > > > Guo
> > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main
> > > driver
> > > > > > behind
> > > > > > > > > > > > FLINK-9641
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on
> > this.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and
> > FLINK-9168
> > > > was
> > > > > > > > created
> > > > > > > > > > for
> > > > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > > > > integration
> > > > > > > was
> > > > > > > > > done
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a year
> > ago
> > > > and
> > > > > we
> > > > > > > > ended
> > > > > > > > > > up
> > > > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar
> users
> > to
> > > > use
> > > > > > > Flink
> > > > > > > > > to
> > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is
> > pretty
> > > > > simple
> > > > > > > and
> > > > > > > > > > there
> > > > > > > > > > > > is
> > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we
> > have
> > > > > made a
> > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > > > > first-class
> > > > > > > > > > citizen in
> > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing
> engines
> > > for
> > > > > > > > processing
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink
> how
> > > to
> > > > > > > > integrate
> > > > > > > > > > with
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> > > pulsar-flink
> > > > > > > > > connectors
> > > > > > > > > > from
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and
> > > catalog
> > > > > API
> > > > > > as
> > > > > > > > the
> > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With that
> > > being
> > > > > > said,
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register
> pulsar
> > > as a
> > > > > > flink
> > > > > > > > > > catalog
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink
> SQL.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example
> about
> > > how
> > > > > to
> > > > > > > use
> > > > > > > > > > Pulsar
> > > > > > > > > > > > as
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also
> written a
> > > > blog
> > > > > > post
> > > > > > > > > > > > explaining
> > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink
> 1.9
> > > and
> > > > > > what
> > > > > > > > are
> > > > > > > > > > the
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is
> not
> > > > just
> > > > > a
> > > > > > > > simple
> > > > > > > > > > data
> > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a
> fully
> > > > > > integrated
> > > > > > > > > > streaming
> > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > > > > schema/catalog
> > > > > > > > > and
> > > > > > > > > > > > > state).
> > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can create
> a
> > > > great
> > > > > > > > > streaming
> > > > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first,
> unified
> > > > data
> > > > > > > > > > processing.
> > > > > > > > > > > > > Since
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> > > > integration
> > > > > to
> > > > > > > > Flink
> > > > > > > > > > here,
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve and
> > > evolve
> > > > > the
> > > > > > > > > > integration
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink and
> > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a
> bit
> > > > more
> > > > > > > > > background
> > > > > > > > > > > > about
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me
> know
> > > > what
> > > > > > are
> > > > > > > > your
> > > > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019 at
> > > 11:54
> > > > > AM
> > > > > > > Yun
> > > > > > > > > > Tang <
> > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that Pulsar
> > > > becomes
> > > > > > > more
> > > > > > > > > and
> > > > > > > > > > more
> > > > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people
> > > willing
> > > > to
> > > > > > > > > > contribute to
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > > > > discussion,
> > > > > > > > would
> > > > > > > > > > you
> > > > > > > > > > > > > please
> > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship
> > between
> > > > this
> > > > > > > > thread
> > > > > > > > > to
> > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink
> [2]
> > > > > > > connector?
> > > > > > > > > > Will the
> > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or totally
> > > > > different
> > > > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > > >
> > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday,
> > September
> > > 3,
> > > > > > 2019
> > > > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> > [hidden email]
> > > <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS]
> > > > Contribute
> > > > > > > > Pulsar
> > > > > > > > > > Flink
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink Community!
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to open
> > the
> > > > > > > discussion
> > > > > > > > > of
> > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back to
> > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
> introduction
> > > to
> > > > > > Apache
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1] is a
> > > > > > > multi-tenant,
> > > > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging
> > system.
> > > > > > Pulsar
> > > > > > > > > > includes
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for
> > > multiple
> > > > > > > > clusters
> > > > > > > > > > in a
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> > geo-replication
> > > of
> > > > > > > > messages
> > > > > > > > > > across
> > > > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end
> latency,
> > > > > > seamless
> > > > > > > > > > > > scalability
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed
> message
> > > > > delivery
> > > > > > > > with
> > > > > > > > > > > > > persistent
> > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > > > > BookKeeper.
> > > > > > > > > > Nowadays,
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> > > companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of
> Pulsar
> > > > Flink
> > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink
> > > connector
> > > > we
> > > > > > are
> > > > > > > > > > planning
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and
> Pulsar
> > > > 2.4.0.
> > > > > > The
> > > > > > > > > main
> > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a
> streaming
> > > > > source
> > > > > > > with
> > > > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming
> > results
> > > to
> > > > > > > Pulsar
> > > > > > > > > with
> > > > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this to
> > > > > > exactly-once
> > > > > > > > as
> > > > > > > > > > well
> > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction features
> > > ready
> > > > in
> > > > > > its
> > > > > > > > > 2.5.0
> > > > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink
> new
> > > > Table
> > > > > > API
> > > > > > > > Type
> > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> > > (de)serialize
> > > > > > > > messages
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with
> Flink
> > > new
> > > > > > > Catalog
> > > > > > > > > API
> > > > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar topics
> as
> > > > > tables
> > > > > > in
> > > > > > > > > > Table API
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Becket Qin]

Hi Stephan,

I have no doubt about the value of adding Pulsar connector to Flink repo.
My concern is about how exactly we are going to do it.

As mentioned before, I believe that we can handle connectors more
> pragmatically and less strict than the core of Flink, if it helps unlocking
> users.

I can see the benefit of being less restrict for the initial connector code
adoption. However, I don't think we should be less restrict on the
maintenance commitment once the code is in Flink repo. It only makes sense
to check in something and ask users to use if we plan to maintain it.

If I understand correctly, the current plan so far is following:
1. release 1.10
   - Check in Pulsar connector on old interface and label it as beta
version.
   - encourage users to try it and report bugs.
2. release 1.11
   - Check in Pulsar connector on new interface (a.k.a new Pulsar
connector) and label it as beta version
   - Deprecate the old Pulsar connector
   - Fix bugs reported on old Pulsar connector from release 1.10
   - Ask users to migrate from old Pulsar connector to new Pulsar connector
3. release 1.12
   - Announce end of support for old Pulsar connector and remove the code
   - Fix bugs reported on new Pulsar connector.

If this is the plan, it seems neither Flink nor the users trying the old
Pulsar connector will benefit from this experimental old Pulsar connector,
because whatever feedbacks we got or bugs we fix on the old Pulsar
connector are immediately thrown away in one or two releases.

If we check in the old Pulsar connector right now, the only option I see is
to maintain it for a while (e.g. a year or more). IMO, the immediate
deprecation and code removal hurts the users much more than asking them to
wait for another release. I personally think that we can avoid this
maintenance burden by going directly to the new Pulsar connector,
especially given that users can still use the connector even if they are
not in Flink repo. That said, I am OK with maintaining both old and new
Pulsar connector if we believe that having the Pulsar connector available
right now in Flink repo is more important.

Thanks,

Jiangjie (Becket) Qin

On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]> wrote:

> My assumption is as Sijie's, that once the connector is either part of
> Flink, or part of the streamnative repo. No double maintenance.
>
> I feel this discussion is very much caught in problems that are all
> solvable if we want to solve them.
> Maybe we can think what our goal for users and the communities is?
>
>   - Do we want to help build a relationship between the Pulsar and Flink
> open source communities?
>   - Will users find a connector in the streamnative repository?
>   - Will users trust a connector that is not part of Flink as much?
>
> And then decide what is best according to the overall goals there.
> As mentioned before, I believe that we can handle connectors more
> pragmatically and less strict than the core of Flink, if it helps unlocking
> users.
>
> Best,
> Stephan
>
>
>
> On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]> wrote:
>
> > Thanks Becket.
> >
> > I think it is better for the Flink community to judge the benefits of
> doing
> > this. I was trying to provide some views from outsiders.
> >
> > Thanks,
> > Sijie
> >
> > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]>
> wrote:
> >
> > > Hi Sijie,
> > >
> > > Yes, we will have to support existing old connectors and new connectors
> > in
> > > parallel for a while. We have to take that maintenance overhead because
> > > existing connectors have been used by the users for a long time. I
> guess
> > It
> > > may take at least a year for us to fully remove the old connectors.
> > >
> > > Process wise, we can do the same for Pulsar connector. But I am not
> sure
> > if
> > > we want to have the same burden on Pulsar connector, and I would like
> to
> > > understand the benefit of doing that.
> > >
> > > For users, the benefit of having the old Pulsar connector checked in
> > seems
> > > limited because 1) that code base will be immediately deprecated in the
> > > next release in 3-4 months; 2) users can always use it even if it is
> not
> > in
> > > the Flink code base. Admittedly it is not as convenient as having it in
> > > Flink code base, but doesn't seem super either. And after 3-4 months,
> > users
> > > can just use the new connector in Flink repo.
> > >
> > > For Flink developers, the old connector code base is not something that
> > we
> > > want to evolve later. Instead, these code will be deprecated and
> > > removed. So why do we want to get a beta version out to attract people
> to
> > > use something we don't want to maintain?
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]> wrote:
> > >
> > > > Thanks everyone here. Sorry for jumping into the discussion here.
> > > >
> > > > I am not very familiar about the deprecation process in Flink. If I
> > > > misunderstood the process, please fix me.
> > > >
> > > > As far as I understand, FLIP-27 is introducing a new unified API for
> > > > connectors. After it introduces the new API
> > > > and before moving all the existing connectors from old API to new
> API,
> > > both
> > > > old ApI and new API will co-exist
> > > > for a while until Flink moves all existing connectors to new API. So
> > the
> > > > Pulsar connector (using old API) can
> > > > follow the deprecation process with other connector using old API and
> > the
> > > > deprecation of old API, no?
> > > >
> > > > If that's the case, I think contributing the current connector back
> to
> > > > Flink rather than maintaining it outside Flink
> > > > would provide a bit more benefits. We can deprecate the existing
> > > > streamnative/pulsar-flink repo and point the users
> > > > to use the connector in Flink repo. So all the review processes will
> > > happen
> > > > within Flink for both old connector and
> > > > new connector. It also reduces the confusions for the users as the
> > > > documentation and code base happen in one place.
> > > >
> > > > Thoughts?
> > > >
> > > > - Sijie
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <[hidden email]>
> > > wrote:
> > > >
> > > > > Thanks for the explanation, Stephan. I have a few questions /
> > thoughts.
> > > > >
> > > > > So that means we will remove the old connector without a major
> > version
> > > > > bump, is that correct?
> > > > >
> > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors
> > will
> > > > > always work because we saw some dependency class collisions in the
> > > past.
> > > > To
> > > > > make it safe we may have to maintain the old code for one more
> > release.
> > > > >
> > > > > To be honest I am still wondering if we have to put the old
> connector
> > > in
> > > > > Flink repo. if we check in the old connector to Flink. We will end
> up
> > > in
> > > > > the following situation:
> > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > > 2. Old connector in Flink Repo, which may be different from the one
> > in
> > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> > > > > 3. New connector in Flink Repo.
> > > > >
> > > > > We need to think about how to make the users in each case happy.
> > > > > - For users of (1), I assume Sijie and Yijie will have to maintain
> > the
> > > > code
> > > > > a bit longer for its own compatibility even after we have (2). In
> > that
> > > > > case, bugs found in old connector may or may not need to be fixed
> in
> > > both
> > > > > Flink and the streamnative/pulsar-flink repo.
> > > > > - For users of (2), will we provide bug fixes? If we do, it will
> be a
> > > > > little awkward because those bug fixes will be immediately
> deprecated
> > > in
> > > > > 1.11, and removed in 1.12. So we are essentially asking users to
> > > migrate
> > > > > away from the bug fix. After Flink 1.12, users may still have to
> > switch
> > > > to
> > > > > use (3) due to the potential dependency class conflicts mentioned
> > > above.
> > > > > - Users of (3) have a much easier life and don't need to worry too
> > > much.
> > > > >
> > > > > The above story seems a little complicated to tell. I think it will
> > be
> > > > much
> > > > > easier to not have (2) at all.
> > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > > 3. New connector in Flink Repo.
> > > > >
> > > > > - Old connector will only be maintained in
> streamnative/pulsar-flink
> > > repo
> > > > > until it is fully deprecated. Users can always use the existing
> > Pulsar
> > > > > connector in that repo.
> > > > > - New connector will be in Flink repo and maintained like the other
> > > > > connectors.
> > > > >
> > > > > This seems much simpler for users understand and they will not be
> > > blocked
> > > > > from using the old connector. If the concern is about the quality
> of
> > > the
> > > > > connector in streamnative/pulsar-flink repo, is it enough for us
> just
> > > to
> > > > > review the code in streamnative/pulsar-flink connector to make sure
> > it
> > > > > looks good from Flink's perspective?
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]>
> > wrote:
> > > > >
> > > > > > My take would be the following:
> > > > > >
> > > > > >   - If we merge the connector now and replace it with a FLIP-27
> > > version
> > > > > > before the 1.10 release, then we need no deprecation process
> > > > > >   - If we don't manage to replace it with a FLIP-27 version
> before
> > > the
> > > > > 1.10
> > > > > > release, than it is good that we have the other version, so no
> > users
> > > > get
> > > > > > blocked.
> > > > > >
> > > > > > In the latter case we can see how we want to do it. Immediate
> > removal
> > > > of
> > > > > > the old version or deprecation label and keeping it for one more
> > > > release.
> > > > > > Given that you should be able to use a Flink 1.10 connector with
> > > Flink
> > > > > 1.11
> > > > > > as well (stable public APIs) there is also a workaround if you
> need
> > > an
> > > > > old
> > > > > > connector in a newer version. So immediate removal might even be
> > > > > feasible.
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Stephan,
> > > > > > >
> > > > > > > Thanks for the clarification. I completely agree with you and
> > > Thomas
> > > > on
> > > > > > the
> > > > > > > process of adding connectors to Flink repo. However, I am
> > wondering
> > > > > what
> > > > > > is
> > > > > > > the deprecation process? Given the main concern here was that
> we
> > > may
> > > > > have
> > > > > > > to maintain two Pulsar connector code bases until the old one
> is
> > > > > removed
> > > > > > > from the repo, it would be good to know how long we have to do
> > > that.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <[hidden email]
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Some quick thoughts on the connector contribution process. I
> > > > > basically
> > > > > > > > reiterate here what Thomas mentioned in another thread about
> > the
> > > > > > Kinesis
> > > > > > > > connector.
> > > > > > > >
> > > > > > > > For connectors, we should favor a low-overhead contribution
> > > > process,
> > > > > > and
> > > > > > > > accept user code and changes more readily than in the core
> > > system.
> > > > > > > > That is because connectors have both a big variety of
> scenarios
> > > > they
> > > > > > get
> > > > > > > > used in (only through use and many small contributions do
> they
> > > > become
> > > > > > > > really useful over time) and at the same time, and committers
> > do
> > > > not
> > > > > > use
> > > > > > > > the connector themselves and usually cannot foresee too well
> > what
> > > > is
> > > > > > > > needed.
> > > > > > > >
> > > > > > > > Further more, a missing connector (or connector feature) is
> > > often a
> > > > > > > bigger
> > > > > > > > show stopper for users than a missing API or system feature.
> > > > > > > >
> > > > > > > > Along these lines of thougt, the conclusion would be to take
> > the
> > > > > Pulsar
> > > > > > > > connector now, focus the review on legal/dependencies/rough
> > code
> > > > > style
> > > > > > > and
> > > > > > > > conventions, label it as "beta" (in the sense of "new code"
> > that
> > > is
> > > > > > "not
> > > > > > > > yet tested through longer use") and go ahead. And then evolve
> > it
> > > > > > quickly
> > > > > > > > without putting formal blockers in the way, meaning also
> > adding a
> > > > new
> > > > > > > FLIP
> > > > > > > > 27 version when it is there.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yijie,
> > > > > > > > >
> > > > > > > > > Could you please follow the FLIP process to start a new
> FLIP
> > > > > > > [DISCUSSION]
> > > > > > > > > thread in the mailing list?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > > > > >
> > > > > > > > > I see two FLIP-69 discussion in the mailing list now. So
> > there
> > > > is a
> > > > > > > FLIP
> > > > > > > > > number collision. Can you change the FLIP number to 72?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yijie,
> > > > > > > > > >
> > > > > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > > > > Would you mind enabling comments/suggestions on the
> google
> > > doc
> > > > > > link?
> > > > > > > > This
> > > > > > > > > > way the contributors from the community can comment on
> the
> > > doc.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Rong
> > > > > > > > > >
> > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello everyone,
> > > > > > > > > > >
> > > > > > > > > > > I've drafted a FLIP that describes the current design
> of
> > > the
> > > > > > Pulsar
> > > > > > > > > > > connector:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > > > > >
> > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Yijie
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi All,
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry for joining the discussion late and thanks
> Yijie
> > &
> > > > > Sijie
> > > > > > > for
> > > > > > > > > > > driving
> > > > > > > > > > > > the discussion.
> > > > > > > > > > > > I also think the Pulsar connector would be a very
> > > valuable
> > > > > > > addition
> > > > > > > > > to
> > > > > > > > > > > > Flink. I can also help out a bit on the review side
> :-)
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding the timeline, I also share concerns with
> > Becket
> > > > on
> > > > > > the
> > > > > > > > > > > > relationship between the new Pulsar connector and
> > > FLIP-27.
> > > > > > > > > > > > There's also another discussion just started by
> Stephan
> > > on
> > > > > > > dropping
> > > > > > > > > > Kafka
> > > > > > > > > > > > 9/10 support on next Flink release [1].  Although the
> > > > > situation
> > > > > > > is
> > > > > > > > > > > somewhat
> > > > > > > > > > > > different, and Kafka 9/10 connector has been in Flink
> > for
> > > > > > almost
> > > > > > > > 3-4
> > > > > > > > > > > years,
> > > > > > > > > > > > based on the discussion I am not sure if a major
> > version
> > > > > > release
> > > > > > > > is a
> > > > > > > > > > > > requirement for removing old connector supports.
> > > > > > > > > > > >
> > > > > > > > > > > > I think there shouldn't be a blocker if we agree the
> > old
> > > > > > > connector
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is there.
> > As
> > > > > > Stephan
> > > > > > > > > > stated,
> > > > > > > > > > > it
> > > > > > > > > > > > is easier to contribute the source sooner and adjust
> it
> > > > > later.
> > > > > > > > > > > > We should also ensure we clearly communicate the
> > message:
> > > > for
> > > > > > > > > example,
> > > > > > > > > > > > putting an experimental flag on the pre-FLIP27
> > connector
> > > > page
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Rong
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Technically speaking, removing the old connector
> code
> > > is
> > > > a
> > > > > > > > > backwards
> > > > > > > > > > > > > incompatible change which requires a major version
> > > bump,
> > > > > i.e.
> > > > > > > > Flink
> > > > > > > > > > > 2.x.
> > > > > > > > > > > > > Given that we don't have a clear plan on when to
> have
> > > the
> > > > > > next
> > > > > > > > > major
> > > > > > > > > > > > > version release, it seems unclear how long the old
> > > > > connector
> > > > > > > code
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > there if we check it in right now. Or will we
> remove
> > > the
> > > > > old
> > > > > > > > > > connector
> > > > > > > > > > > > > without a major version bump? In any case, it
> sounds
> > > not
> > > > > > quite
> > > > > > > > user
> > > > > > > > > > > > > friendly to the those who might use the old Pulsar
> > > > > > connector. I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > > sure
> > > > > > > > > > > > > if it is worth these potential problems in order to
> > > have
> > > > > the
> > > > > > > > Pulsar
> > > > > > > > > > > source
> > > > > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Agreed, if we check in the old code, we should
> make
> > > it
> > > > > > clear
> > > > > > > > that
> > > > > > > > > > it
> > > > > > > > > > > will
> > > > > > > > > > > > > > be removed as soon as the FLIP-27 based version
> of
> > > the
> > > > > > > > connector
> > > > > > > > > is
> > > > > > > > > > > > > there.
> > > > > > > > > > > > > > We should not commit to maintaining the old
> > version,
> > > > that
> > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > indeed
> > > > > > > > > > > > > > too much overhead.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Yes, the overhead would just be review
> capacity.
> > In
> > > > > > fact, I
> > > > > > > > am
> > > > > > > > > > not
> > > > > > > > > > > > > > worrying
> > > > > > > > > > > > > > > too much about the review capacity. That is
> just
> > a
> > > > one
> > > > > > time
> > > > > > > > > cost.
> > > > > > > > > > > My
> > > > > > > > > > > > > > > concern is mainly about the long term burden.
> > > Assume
> > > > we
> > > > > > > have
> > > > > > > > > new
> > > > > > > > > > > source
> > > > > > > > > > > > > > > interface ready in 1.10 with newly added Pulsar
> > > > > > connectors
> > > > > > > in
> > > > > > > > > old
> > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to new
> > > > source
> > > > > > > > > interface,
> > > > > > > > > > > the
> > > > > > > > > > > > > old
> > > > > > > > > > > > > > > Pulsar connector might be deprecated almost
> > > > immediately
> > > > > > > after
> > > > > > > > > > > checked
> > > > > > > > > > > > > in,
> > > > > > > > > > > > > > > but we may still have to maintain two code
> bases.
> > > For
> > > > > the
> > > > > > > > > > existing
> > > > > > > > > > > > > > > connectors, we have to do that anyways. But it
> > > would
> > > > be
> > > > > > > good
> > > > > > > > to
> > > > > > > > > > > avoid
> > > > > > > > > > > > > > > introducing a new connector with the same
> > problem.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Nice to see this lively discussion about the
> > > Pulsar
> > > > > > > > > connector.
> > > > > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a
> > community
> > > > > > package
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Looks like the discussion is more going
> towards
> > > > > > > > > contribution. I
> > > > > > > > > > > think
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > is good, especially if we think that we want
> to
> > > > > build a
> > > > > > > > > > similarly
> > > > > > > > > > > > > deep
> > > > > > > > > > > > > > > > integration with Pulsar as we have for
> example
> > > with
> > > > > > > Kafka.
> > > > > > > > > The
> > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > already looks like a more thorough connector
> > than
> > > > > many
> > > > > > > > others
> > > > > > > > > > we
> > > > > > > > > > > have
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the repository.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > With either a repo split, or the new build
> > > system,
> > > > I
> > > > > > hope
> > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Becket offered some help already, I can also
> > > help a
> > > > > > bit.
> > > > > > > I
> > > > > > > > > hope
> > > > > > > > > > > that
> > > > > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making
> some
> > > > > PoC-ing
> > > > > > > > > > > progress, but
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > take 2 more months, I would estimate, before
> it
> > > is
> > > > > > fully
> > > > > > > > > > > available.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If we want to be on the safe side with the
> > > > > > contribution,
> > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > > > > > contribute the source sooner and adjust it
> > later.
> > > > > That
> > > > > > > > would
> > > > > > > > > > also
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > in case things get crazy towards the 1.10
> > feature
> > > > > > freeze
> > > > > > > > and
> > > > > > > > > it
> > > > > > > > > > > would
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > hard to find time to review the new changes.
> > > > > > > > > > > > > > > > What would be the overhead of contributing
> now?
> > > > Given
> > > > > > > that
> > > > > > > > > the
> > > > > > > > > > > code
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > already there, it looks like it would be only
> > > > review
> > > > > > > > > capacity,
> > > > > > > > > > > right?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie Shen <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your attention and the promotion
> > of
> > > > this
> > > > > > > work.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as possible
> > for
> > > > more
> > > > > > > > > specific
> > > > > > > > > > > > > > > discussions.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not
> > reached
> > > a
> > > > > > > > consensus.
> > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > I will explain all the functionalities of
> the
> > > > > > existing
> > > > > > > > > > > connector in
> > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and
> > Catalog)
> > > to
> > > > > > > > continue
> > > > > > > > > > our
> > > > > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket Qin
> <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If we agree that the goal is to have
> Pulsar
> > > > > > connector
> > > > > > > > in
> > > > > > > > > > > 1.10,
> > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector
> to
> > > > Flink
> > > > > > main
> > > > > > > > > repo
> > > > > > > > > > > as it
> > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink right
> > away
> > > > as
> > > > > > > there
> > > > > > > > is
> > > > > > > > > > no
> > > > > > > > > > > > > change
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink
> 1.10
> > > is
> > > > > > going
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > code
> > > > > > > > > > > > > > > freeze
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > late Nov and let's say we give a month to
> > the
> > > > > > > > development
> > > > > > > > > > and
> > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Pulsar connector, we need to have FLIP-27
> > by
> > > > late
> > > > > > > Oct.
> > > > > > > > > > There
> > > > > > > > > > > are
> > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > > > > weeks. Personally I think it is doable.
> If
> > > > > FLIP-27
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > ready
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > late
> > > > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar
> > > > connector
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > interface. This means we will have Pulsar
> > > > > connector
> > > > > > > in
> > > > > > > > > > Flink
> > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > either
> > > > > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Because we are going to have Pulsar sink
> > and
> > > > > source
> > > > > > > > > checked
> > > > > > > > > > > in
> > > > > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > > > > it might make sense to have two FLIPs,
> one
> > > for
> > > > > > Pulsar
> > > > > > > > > sink
> > > > > > > > > > > and
> > > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > Pulsar source. And we can start the work
> on
> > > > > Pulsar
> > > > > > > sink
> > > > > > > > > > right
> > > > > > > > > > > > > away.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie Guo
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > What's the take from Flink community?
> > Shall
> > > > we
> > > > > > wait
> > > > > > > > for
> > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > > > > proceed to next steps? And what the
> next
> > > > steps
> > > > > > are?
> > > > > > > > :-)
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen
> Li <
> > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector in
> > > Flink
> > > > > can
> > > > > > > be a
> > > > > > > > > > good
> > > > > > > > > > > > > mutual
> > > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar
> > > > connector
> > > > > is
> > > > > > > the
> > > > > > > > > 1st
> > > > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > that integrates with Flink's metadata
> > > > > > management
> > > > > > > > > system
> > > > > > > > > > > and
> > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > > > > It'll be cool to see how the
> > integration
> > > > > turns
> > > > > > > out
> > > > > > > > > and
> > > > > > > > > > > > > whether
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which
> are
> > > > > > currently
> > > > > > > in
> > > > > > > > > > > Beta, to
> > > > > > > > > > > > > > > cater
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in
> > favor
> > > of
> > > > > > > merging
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized
> > PRs,
> > > > e.g.
> > > > > > > maybe
> > > > > > > > > one
> > > > > > > > > > > for
> > > > > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > > > > source/sink functionalities and
> another
> > > for
> > > > > > > schema
> > > > > > > > > and
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for
> > > > FLIP-27.
> > > > > > But
> > > > > > > I
> > > > > > > > > > don't
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > > > > should be a blocker in cases where it
> > > > cannot
> > > > > > make
> > > > > > > > its
> > > > > > > > > > way
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of
> time
> > > for
> > > > > > > > > committers
> > > > > > > > > > to
> > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to
> new
> > > > > > > interfaces.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM Becket
> > > Qin <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > You are right. It all depends on
> when
> > > the
> > > > > new
> > > > > > > > > source
> > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it
> would
> > > be
> > > > > > there
> > > > > > > in
> > > > > > > > > > > about a
> > > > > > > > > > > > > > month
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It would
> > > also
> > > > be
> > > > > > > good
> > > > > > > > to
> > > > > > > > > > > hear
> > > > > > > > > > > > > what
> > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > > > > and Stephan think as they are also
> > > > involved
> > > > > > in
> > > > > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > In general I think we should have
> > > Pulsar
> > > > > > > > connector
> > > > > > > > > in
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > > > > preferably with the new source
> > > interface.
> > > > > We
> > > > > > > can
> > > > > > > > > also
> > > > > > > > > > > check
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > > > > with old source interface, but I
> > > suspect
> > > > > few
> > > > > > > > users
> > > > > > > > > > > will use
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > next official release. Therefore,
> it
> > > > seems
> > > > > > > > > reasonable
> > > > > > > > > > > to
> > > > > > > > > > > > > > wait a
> > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > to see whether we can jump to the
> new
> > > > > source
> > > > > > > > > > > interface. As
> > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a
> > > little
> > > > > bit
> > > > > > > > > doesn't
> > > > > > > > > > > seem
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > hurt
> > > > > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM Till
> > > > > Rohrmann
> > > > > > <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem
> > would
> > > be
> > > > > if
> > > > > > we
> > > > > > > > > > > committed
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > connector before the new source
> > > > interface
> > > > > > is
> > > > > > > > > ready.
> > > > > > > > > > > If I
> > > > > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > correctly, then we need to
> support
> > > the
> > > > > old
> > > > > > > > source
> > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > existing connectors. By checking
> it
> > > in
> > > > > > early
> > > > > > > I
> > > > > > > > > > could
> > > > > > > > > > > see
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > our users could start using the
> > > > connector
> > > > > > > > > earlier.
> > > > > > > > > > > > > > Moreover,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar
> integration
> > > is
> > > > > > being
> > > > > > > > > > delayed
> > > > > > > > > > > in
> > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > > > source interface should be
> delayed.
> > > The
> > > > > > only
> > > > > > > > > > > downside I
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > > > > review effort and potential fixes
> > > which
> > > > > > might
> > > > > > > > be
> > > > > > > > > > > > > irrelevant
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > source interface implementation.
> I
> > > > guess
> > > > > it
> > > > > > > > > mainly
> > > > > > > > > > > > > depends
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > > > > we are when the new source
> > interface
> > > > will
> > > > > > be
> > > > > > > > > ready.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM
> > Becket
> > > > > Qin <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your
> thoughts.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Just want to have some update
> on
> > > > > FLIP-27.
> > > > > > > > > > Although
> > > > > > > > > > > the
> > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > discussion thread has been
> quiet
> > > for
> > > > > some
> > > > > > > > > time, a
> > > > > > > > > > > few
> > > > > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > > > > contributors in Flink community
> > > were
> > > > > > > actually
> > > > > > > > > > > > > prototyping
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > > > > We have made some good progress
> > > there
> > > > > but
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > > update
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > > > > after the entire thing is
> > verified
> > > to
> > > > > > work
> > > > > > > in
> > > > > > > > > > case
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > > > > minute surprise in the
> > > > implementation.
> > > > > I
> > > > > > > > don't
> > > > > > > > > > > have an
> > > > > > > > > > > > > > > exact
> > > > > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > > > > guess it is going to be within
> a
> > > > month
> > > > > or
> > > > > > > so.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I am happy to review the
> current
> > > > Flink
> > > > > > > Pulsar
> > > > > > > > > > > connector
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be
> good
> > to
> > > > > avoid
> > > > > > > the
> > > > > > > > > > case
> > > > > > > > > > > that
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some
> review
> > > > > efforts
> > > > > > > and
> > > > > > > > > > > shortly
> > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39 AM
> > > Yijie
> > > > > > Shen <
> > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback
> and
> > > > > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of
> the
> > > > > > connector
> > > > > > > > has
> > > > > > > > > > > always
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > users with the latest
> features
> > of
> > > > > both
> > > > > > > > > systems
> > > > > > > > > > as
> > > > > > > > > > > > > soon
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the
> > > connector
> > > > > to
> > > > > > > > Flink
> > > > > > > > > > and
> > > > > > > > > > > hope
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback from
> > > Flink
> > > > > > > experts
> > > > > > > > > to
> > > > > > > > > > > ensure
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its
> > > > existence
> > > > > > at
> > > > > > > > the
> > > > > > > > > > > > > beginning
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > > > > the connector implementation
> > > based
> > > > on
> > > > > > > Flink
> > > > > > > > > > 1.9;
> > > > > > > > > > > we
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > build a connector that
> supports
> > > > both
> > > > > > > batch
> > > > > > > > > and
> > > > > > > > > > > stream
> > > > > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > > > > However, it has been inactive
> > for
> > > > > some
> > > > > > > > time,
> > > > > > > > > so
> > > > > > > > > > > we
> > > > > > > > > > > > > > > decided
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > a connector with most of the
> > new
> > > > > > > features,
> > > > > > > > > such
> > > > > > > > > > > as
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API
> first.
> > We
> > > > > will
> > > > > > > pay
> > > > > > > > > > > attention
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> > > > > incorporate
> > > > > > it
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status of
> > the
> > > > > > > connector,
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> > > > repository
> > > > > > and
> > > > > > > > > aimed
> > > > > > > > > > to
> > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are
> also
> > > > happy
> > > > > to
> > > > > > > > hear
> > > > > > > > > > > > > > suggestions
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > supervision from the Flink
> > > > community
> > > > > to
> > > > > > > > > improve
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > > > > performance of the connector
> > > > > > > continuously.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59
> AM
> > > > Sijie
> > > > > > Guo
> > > > > > > <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the
> > > comments
> > > > > and
> > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the
> main
> > > > > question
> > > > > > > > here
> > > > > > > > > is
> > > > > > > > > > > > > about -
> > > > > > > > > > > > > > > > "how
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > community maintain the
> > > > connector?".
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts from
> > > > myself.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where to
> > > host
> > > > > this
> > > > > > > > > > > integration
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > kind
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there can
> be
> > > many
> > > > > > ways
> > > > > > > to
> > > > > > > > > > > achieve
> > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > > As part of the
> contribution,
> > > what
> > > > > we
> > > > > > > are
> > > > > > > > > > > looking
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > here
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > > > > communities can build the
> > > > > > collaboration
> > > > > > > > > > > > > relationship
> > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > > > > the integration between
> > Pulsar
> > > > and
> > > > > > > Flink.
> > > > > > > > > > Even
> > > > > > > > > > > we
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in Flink
> > > > > > community.
> > > > > > > We
> > > > > > > > > are
> > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we
> have
> > > less
> > > > > > > > > experiences
> > > > > > > > > > > in
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > community. In order to make
> > > sure
> > > > we
> > > > > > > > > maintain
> > > > > > > > > > > and
> > > > > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > > > > a high-quality pulsar-flink
> > > > > > integration
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > > who
> > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need some
> > help
> > > > > from
> > > > > > > the
> > > > > > > > > > > experts
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following
> > > FLIP-27
> > > > > > for a
> > > > > > > > > > while.
> > > > > > > > > > > > > > > Originally
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > > > > of contributing the
> > connectors
> > > > back
> > > > > > > after
> > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in
> > FLIP-27.
> > > > But
> > > > > we
> > > > > > > > > decided
> > > > > > > > > > > to
> > > > > > > > > > > > > > > initiate
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > > > > as early as possible.
> Because
> > > we
> > > > > > > believe
> > > > > > > > > > there
> > > > > > > > > > > are
> > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > > > > it now rather than later.
> As
> > > part
> > > > > of
> > > > > > > > > > > contribution,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > community understand more
> > about
> > > > > > Pulsar
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help Flink
> > > > > community
> > > > > > > > > verify
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g.
> > catalog
> > > > > API).
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at 5:24
> > AM
> > > > > Becket
> > > > > > > > Qin <
> > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest
> in
> > > > > > > contributing
> > > > > > > > > the
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think
> having
> > > > Pulsar
> > > > > > > > > connector
> > > > > > > > > > > with
> > > > > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to
> Flink.
> > > So
> > > > I
> > > > > am
> > > > > > > > happy
> > > > > > > > > > the
> > > > > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also
> > like
> > > to
> > > > > > > provide
> > > > > > > > > > some
> > > > > > > > > > > > > > context
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors
> > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink
> > > maintains
> > > > > its
> > > > > > > > > > > connector has
> > > > > > > > > > > > > > hit
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > > > > With more and more
> > connectors
> > > > > > coming
> > > > > > > > into
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > repo,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long
> build
> > > and
> > > > > > > testing
> > > > > > > > > > > time. To
> > > > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do the
> > > > > following:
> > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the
> connectors
> > > > into
> > > > > a
> > > > > > > > > separate
> > > > > > > > > > > > > > > repository.
> > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to potential
> > > > solution
> > > > > > to
> > > > > > > > > > shorten
> > > > > > > > > > > the
> > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the
> connectors
> > > to
> > > > > stay
> > > > > > > as
> > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > > > > to provide good support
> for
> > > > > > > > functionality
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create a
> > Flink
> > > > > > > Ecosystem
> > > > > > > > > > > project
> > > > > > > > > > > > > > > website
> > > > > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > > > > through some final
> approval
> > > > > > process.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Given the above efforts,
> it
> > > > would
> > > > > > be
> > > > > > > > > great
> > > > > > > > > > to
> > > > > > > > > > > > > first
> > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an
> > > > ecosystem
> > > > > > > > project
> > > > > > > > > > with
> > > > > > > > > > > > > great
> > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the
> Flink
> > > > Pulsar
> > > > > > > > > connector
> > > > > > > > > > > is
> > > > > > > > > > > > > > tested
> > > > > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > we can learn something to
> > > > > maintain
> > > > > > it
> > > > > > > > as
> > > > > > > > > an
> > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > > > > quality and test
> coverage.
> > If
> > > > the
> > > > > > > > quality
> > > > > > > > > > as
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as
> > well
> > > > > adopt
> > > > > > it
> > > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > main
> > > > > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing
> effort
> > > is
> > > > > > > FLIP-27
> > > > > > > > > > where
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > > > > architecture
> > > > > > > and
> > > > > > > > > > > > > interface.
> > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore
> > > timing
> > > > > > wise,
> > > > > > > if
> > > > > > > > > we
> > > > > > > > > > > are
> > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > connector in main repo, I
> > am
> > > > > > > wondering
> > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar connector
> > > adapt
> > > > to
> > > > > > the
> > > > > > > > new
> > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> 4:32
> > > PM
> > > > > > > Chesnay
> > > > > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried that
> we
> > > may
> > > > > end
> > > > > > > up
> > > > > > > > > > > repeating
> > > > > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2
> > > attempts
> > > > > at
> > > > > > > > > > > contributing a
> > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed because
> > no
> > > > > > > committer
> > > > > > > > > was
> > > > > > > > > > > > > getting
> > > > > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a
> > > > dedicated
> > > > > > > > > > discussion
> > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and getting
> > > > several
> > > > > > +1's
> > > > > > > > > from
> > > > > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > We should really make
> > sure
> > > > that
> > > > > > if
> > > > > > > we
> > > > > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it will
> > > actually
> > > > > get
> > > > > > > the
> > > > > > > > > > > attention
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined
> to
> > > > > > recommend
> > > > > > > > > > > maintaining
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link to
> > it
> > > > from
> > > > > > the
> > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming page
> > for
> > > > > > sharing
> > > > > > > > > > > artifacts
> > > > > > > > > > > > > > among
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that
> > anyway?),
> > > > > this
> > > > > > > may
> > > > > > > > > be a
> > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16,
> Till
> > > > > > Rohrmann
> > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for
> > starting
> > > > > this
> > > > > > > > > > discussion
> > > > > > > > > > > > > > Yijie.
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would be a
> > very
> > > > > > > valuable
> > > > > > > > > > > addition
> > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it
> > would
> > > > > > further
> > > > > > > > > > expand
> > > > > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project
> > > perspective
> > > > it
> > > > > > > makes
> > > > > > > > > > > sense for
> > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream
> project.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> > concern/question
> > > is
> > > > > how
> > > > > > > can
> > > > > > > > > the
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have
> seen
> > > in
> > > > > the
> > > > > > > past
> > > > > > > > > > that
> > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > > > > actively developed
> > > > components
> > > > > > > > because
> > > > > > > > > > > they
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > external system and
> > with
> > > > > Flink.
> > > > > > > > Given
> > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
> > maintaining,
> > > > > > > improving
> > > > > > > > > and
> > > > > > > > > > > > > > evolving
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we
> can
> > > > > achieve
> > > > > > > > this.
> > > > > > > > > > > Hence,
> > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019
> at
> > > 2:03
> > > > > AM
> > > > > > > > Sijie
> > > > > > > > > > Guo
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the main
> > > > driver
> > > > > > > behind
> > > > > > > > > > > > > FLINK-9641
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context on
> > > this.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and
> > > FLINK-9168
> > > > > was
> > > > > > > > > created
> > > > > > > > > > > for
> > > > > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink. The
> > > > > > integration
> > > > > > > > was
> > > > > > > > > > done
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a
> year
> > > ago
> > > > > and
> > > > > > we
> > > > > > > > > ended
> > > > > > > > > > > up
> > > > > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar
> > users
> > > to
> > > > > use
> > > > > > > > Flink
> > > > > > > > > > to
> > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is
> > > pretty
> > > > > > simple
> > > > > > > > and
> > > > > > > > > > > there
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year, we
> > > have
> > > > > > made a
> > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as the
> > > > > > first-class
> > > > > > > > > > > citizen in
> > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing
> > engines
> > > > for
> > > > > > > > > processing
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to rethink
> > how
> > > > to
> > > > > > > > > integrate
> > > > > > > > > > > with
> > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> > > > pulsar-flink
> > > > > > > > > > connectors
> > > > > > > > > > > from
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API and
> > > > catalog
> > > > > > API
> > > > > > > as
> > > > > > > > > the
> > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With
> that
> > > > being
> > > > > > > said,
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register
> > pulsar
> > > > as a
> > > > > > > flink
> > > > > > > > > > > catalog
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using Flink
> > SQL.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example
> > about
> > > > how
> > > > > > to
> > > > > > > > use
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also
> > written a
> > > > > blog
> > > > > > > post
> > > > > > > > > > > > > explaining
> > > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with Flink
> > 1.9
> > > > and
> > > > > > > what
> > > > > > > > > are
> > > > > > > > > > > the
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar is
> > not
> > > > > just
> > > > > > a
> > > > > > > > > simple
> > > > > > > > > > > data
> > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a
> > fully
> > > > > > > integrated
> > > > > > > > > > > streaming
> > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, source,
> > > > > > > > schema/catalog
> > > > > > > > > > and
> > > > > > > > > > > > > > state).
> > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can
> create
> > a
> > > > > great
> > > > > > > > > > streaming
> > > > > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first,
> > unified
> > > > > data
> > > > > > > > > > > processing.
> > > > > > > > > > > > > > Since
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> > > > > integration
> > > > > > to
> > > > > > > > > Flink
> > > > > > > > > > > here,
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve
> and
> > > > evolve
> > > > > > the
> > > > > > > > > > > integration
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink
> and
> > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give you a
> > bit
> > > > > more
> > > > > > > > > > background
> > > > > > > > > > > > > about
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let me
> > know
> > > > > what
> > > > > > > are
> > > > > > > > > your
> > > > > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3, 2019
> at
> > > > 11:54
> > > > > > AM
> > > > > > > > Yun
> > > > > > > > > > > Tang <
> > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that
> Pulsar
> > > > > becomes
> > > > > > > > more
> > > > > > > > > > and
> > > > > > > > > > > more
> > > > > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more people
> > > > willing
> > > > > to
> > > > > > > > > > > contribute to
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any further
> > > > > > discussion,
> > > > > > > > > would
> > > > > > > > > > > you
> > > > > > > > > > > > > > please
> > > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship
> > > between
> > > > > this
> > > > > > > > > thread
> > > > > > > > > > to
> > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and sink
> > [2]
> > > > > > > > connector?
> > > > > > > > > > > Will the
> > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or
> totally
> > > > > > different
> > > > > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > > > >
> > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday,
> > > September
> > > > 3,
> > > > > > > 2019
> > > > > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> > > [hidden email]
> > > > <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: [DISCUSS]
> > > > > Contribute
> > > > > > > > > Pulsar
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink
> Community!
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to
> open
> > > the
> > > > > > > > discussion
> > > > > > > > > > of
> > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0] back
> to
> > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
> > introduction
> > > > to
> > > > > > > Apache
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1]
> is a
> > > > > > > > multi-tenant,
> > > > > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging
> > > system.
> > > > > > > Pulsar
> > > > > > > > > > > includes
> > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support for
> > > > multiple
> > > > > > > > > clusters
> > > > > > > > > > > in a
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> > > geo-replication
> > > > of
> > > > > > > > > messages
> > > > > > > > > > > across
> > > > > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end
> > latency,
> > > > > > > seamless
> > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed
> > message
> > > > > > delivery
> > > > > > > > > with
> > > > > > > > > > > > > > persistent
> > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by Apache
> > > > > > BookKeeper.
> > > > > > > > > > > Nowadays,
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> > > > companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of
> > Pulsar
> > > > > Flink
> > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink
> > > > connector
> > > > > we
> > > > > > > are
> > > > > > > > > > > planning
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and
> > Pulsar
> > > > > 2.4.0.
> > > > > > > The
> > > > > > > > > > main
> > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a
> > streaming
> > > > > > source
> > > > > > > > with
> > > > > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming
> > > results
> > > > to
> > > > > > > > Pulsar
> > > > > > > > > > with
> > > > > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this
> to
> > > > > > > exactly-once
> > > > > > > > > as
> > > > > > > > > > > well
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction
> features
> > > > ready
> > > > > in
> > > > > > > its
> > > > > > > > > > 2.5.0
> > > > > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon Flink
> > new
> > > > > Table
> > > > > > > API
> > > > > > > > > Type
> > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> > > > (de)serialize
> > > > > > > > > messages
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with
> > Flink
> > > > new
> > > > > > > > Catalog
> > > > > > > > > > API
> > > > > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar
> topics
> > as
> > > > > > tables
> > > > > > > in
> > > > > > > > > > > Table API
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Stephan Ewen]

Okay, I see your point, Becket.

Then let us prominently link the Pulsar connector from the Flink connector
docs then, so that users can find it easily.

As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute a
new connector.

On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]> wrote:

> Hi Stephan,
>
> I have no doubt about the value of adding Pulsar connector to Flink repo.
> My concern is about how exactly we are going to do it.
>
> As mentioned before, I believe that we can handle connectors more
> > pragmatically and less strict than the core of Flink, if it helps
> unlocking
> > users.
>
> I can see the benefit of being less restrict for the initial connector code
> adoption. However, I don't think we should be less restrict on the
> maintenance commitment once the code is in Flink repo. It only makes sense
> to check in something and ask users to use if we plan to maintain it.
>
> If I understand correctly, the current plan so far is following:
> 1. release 1.10
>    - Check in Pulsar connector on old interface and label it as beta
> version.
>    - encourage users to try it and report bugs.
> 2. release 1.11
>    - Check in Pulsar connector on new interface (a.k.a new Pulsar
> connector) and label it as beta version
>    - Deprecate the old Pulsar connector
>    - Fix bugs reported on old Pulsar connector from release 1.10
>    - Ask users to migrate from old Pulsar connector to new Pulsar connector
> 3. release 1.12
>    - Announce end of support for old Pulsar connector and remove the code
>    - Fix bugs reported on new Pulsar connector.
>
> If this is the plan, it seems neither Flink nor the users trying the old
> Pulsar connector will benefit from this experimental old Pulsar connector,
> because whatever feedbacks we got or bugs we fix on the old Pulsar
> connector are immediately thrown away in one or two releases.
>
> If we check in the old Pulsar connector right now, the only option I see is
> to maintain it for a while (e.g. a year or more). IMO, the immediate
> deprecation and code removal hurts the users much more than asking them to
> wait for another release. I personally think that we can avoid this
> maintenance burden by going directly to the new Pulsar connector,
> especially given that users can still use the connector even if they are
> not in Flink repo. That said, I am OK with maintaining both old and new
> Pulsar connector if we believe that having the Pulsar connector available
> right now in Flink repo is more important.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]> wrote:
>
> > My assumption is as Sijie's, that once the connector is either part of
> > Flink, or part of the streamnative repo. No double maintenance.
> >
> > I feel this discussion is very much caught in problems that are all
> > solvable if we want to solve them.
> > Maybe we can think what our goal for users and the communities is?
> >
> >   - Do we want to help build a relationship between the Pulsar and Flink
> > open source communities?
> >   - Will users find a connector in the streamnative repository?
> >   - Will users trust a connector that is not part of Flink as much?
> >
> > And then decide what is best according to the overall goals there.
> > As mentioned before, I believe that we can handle connectors more
> > pragmatically and less strict than the core of Flink, if it helps
> unlocking
> > users.
> >
> > Best,
> > Stephan
> >
> >
> >
> > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]> wrote:
> >
> > > Thanks Becket.
> > >
> > > I think it is better for the Flink community to judge the benefits of
> > doing
> > > this. I was trying to provide some views from outsiders.
> > >
> > > Thanks,
> > > Sijie
> > >
> > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]>
> > wrote:
> > >
> > > > Hi Sijie,
> > > >
> > > > Yes, we will have to support existing old connectors and new
> connectors
> > > in
> > > > parallel for a while. We have to take that maintenance overhead
> because
> > > > existing connectors have been used by the users for a long time. I
> > guess
> > > It
> > > > may take at least a year for us to fully remove the old connectors.
> > > >
> > > > Process wise, we can do the same for Pulsar connector. But I am not
> > sure
> > > if
> > > > we want to have the same burden on Pulsar connector, and I would like
> > to
> > > > understand the benefit of doing that.
> > > >
> > > > For users, the benefit of having the old Pulsar connector checked in
> > > seems
> > > > limited because 1) that code base will be immediately deprecated in
> the
> > > > next release in 3-4 months; 2) users can always use it even if it is
> > not
> > > in
> > > > the Flink code base. Admittedly it is not as convenient as having it
> in
> > > > Flink code base, but doesn't seem super either. And after 3-4 months,
> > > users
> > > > can just use the new connector in Flink repo.
> > > >
> > > > For Flink developers, the old connector code base is not something
> that
> > > we
> > > > want to evolve later. Instead, these code will be deprecated and
> > > > removed. So why do we want to get a beta version out to attract
> people
> > to
> > > > use something we don't want to maintain?
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > >
> > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]>
> wrote:
> > > >
> > > > > Thanks everyone here. Sorry for jumping into the discussion here.
> > > > >
> > > > > I am not very familiar about the deprecation process in Flink. If I
> > > > > misunderstood the process, please fix me.
> > > > >
> > > > > As far as I understand, FLIP-27 is introducing a new unified API
> for
> > > > > connectors. After it introduces the new API
> > > > > and before moving all the existing connectors from old API to new
> > API,
> > > > both
> > > > > old ApI and new API will co-exist
> > > > > for a while until Flink moves all existing connectors to new API.
> So
> > > the
> > > > > Pulsar connector (using old API) can
> > > > > follow the deprecation process with other connector using old API
> and
> > > the
> > > > > deprecation of old API, no?
> > > > >
> > > > > If that's the case, I think contributing the current connector back
> > to
> > > > > Flink rather than maintaining it outside Flink
> > > > > would provide a bit more benefits. We can deprecate the existing
> > > > > streamnative/pulsar-flink repo and point the users
> > > > > to use the connector in Flink repo. So all the review processes
> will
> > > > happen
> > > > > within Flink for both old connector and
> > > > > new connector. It also reduces the confusions for the users as the
> > > > > documentation and code base happen in one place.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > - Sijie
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Thanks for the explanation, Stephan. I have a few questions /
> > > thoughts.
> > > > > >
> > > > > > So that means we will remove the old connector without a major
> > > version
> > > > > > bump, is that correct?
> > > > > >
> > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 connectors
> > > will
> > > > > > always work because we saw some dependency class collisions in
> the
> > > > past.
> > > > > To
> > > > > > make it safe we may have to maintain the old code for one more
> > > release.
> > > > > >
> > > > > > To be honest I am still wondering if we have to put the old
> > connector
> > > > in
> > > > > > Flink repo. if we check in the old connector to Flink. We will
> end
> > up
> > > > in
> > > > > > the following situation:
> > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > > > 2. Old connector in Flink Repo, which may be different from the
> one
> > > in
> > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in 1.12)
> > > > > > 3. New connector in Flink Repo.
> > > > > >
> > > > > > We need to think about how to make the users in each case happy.
> > > > > > - For users of (1), I assume Sijie and Yijie will have to
> maintain
> > > the
> > > > > code
> > > > > > a bit longer for its own compatibility even after we have (2). In
> > > that
> > > > > > case, bugs found in old connector may or may not need to be fixed
> > in
> > > > both
> > > > > > Flink and the streamnative/pulsar-flink repo.
> > > > > > - For users of (2), will we provide bug fixes? If we do, it will
> > be a
> > > > > > little awkward because those bug fixes will be immediately
> > deprecated
> > > > in
> > > > > > 1.11, and removed in 1.12. So we are essentially asking users to
> > > > migrate
> > > > > > away from the bug fix. After Flink 1.12, users may still have to
> > > switch
> > > > > to
> > > > > > use (3) due to the potential dependency class conflicts mentioned
> > > > above.
> > > > > > - Users of (3) have a much easier life and don't need to worry
> too
> > > > much.
> > > > > >
> > > > > > The above story seems a little complicated to tell. I think it
> will
> > > be
> > > > > much
> > > > > > easier to not have (2) at all.
> > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > > > 3. New connector in Flink Repo.
> > > > > >
> > > > > > - Old connector will only be maintained in
> > streamnative/pulsar-flink
> > > > repo
> > > > > > until it is fully deprecated. Users can always use the existing
> > > Pulsar
> > > > > > connector in that repo.
> > > > > > - New connector will be in Flink repo and maintained like the
> other
> > > > > > connectors.
> > > > > >
> > > > > > This seems much simpler for users understand and they will not be
> > > > blocked
> > > > > > from using the old connector. If the concern is about the quality
> > of
> > > > the
> > > > > > connector in streamnative/pulsar-flink repo, is it enough for us
> > just
> > > > to
> > > > > > review the code in streamnative/pulsar-flink connector to make
> sure
> > > it
> > > > > > looks good from Flink's perspective?
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]>
> > > wrote:
> > > > > >
> > > > > > > My take would be the following:
> > > > > > >
> > > > > > >   - If we merge the connector now and replace it with a FLIP-27
> > > > version
> > > > > > > before the 1.10 release, then we need no deprecation process
> > > > > > >   - If we don't manage to replace it with a FLIP-27 version
> > before
> > > > the
> > > > > > 1.10
> > > > > > > release, than it is good that we have the other version, so no
> > > users
> > > > > get
> > > > > > > blocked.
> > > > > > >
> > > > > > > In the latter case we can see how we want to do it. Immediate
> > > removal
> > > > > of
> > > > > > > the old version or deprecation label and keeping it for one
> more
> > > > > release.
> > > > > > > Given that you should be able to use a Flink 1.10 connector
> with
> > > > Flink
> > > > > > 1.11
> > > > > > > as well (stable public APIs) there is also a workaround if you
> > need
> > > > an
> > > > > > old
> > > > > > > connector in a newer version. So immediate removal might even
> be
> > > > > > feasible.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
> > [hidden email]>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Stephan,
> > > > > > > >
> > > > > > > > Thanks for the clarification. I completely agree with you and
> > > > Thomas
> > > > > on
> > > > > > > the
> > > > > > > > process of adding connectors to Flink repo. However, I am
> > > wondering
> > > > > > what
> > > > > > > is
> > > > > > > > the deprecation process? Given the main concern here was that
> > we
> > > > may
> > > > > > have
> > > > > > > > to maintain two Pulsar connector code bases until the old one
> > is
> > > > > > removed
> > > > > > > > from the repo, it would be good to know how long we have to
> do
> > > > that.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jiangjie (Becket) Qin
> > > > > > > >
> > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Some quick thoughts on the connector contribution process.
> I
> > > > > > basically
> > > > > > > > > reiterate here what Thomas mentioned in another thread
> about
> > > the
> > > > > > > Kinesis
> > > > > > > > > connector.
> > > > > > > > >
> > > > > > > > > For connectors, we should favor a low-overhead contribution
> > > > > process,
> > > > > > > and
> > > > > > > > > accept user code and changes more readily than in the core
> > > > system.
> > > > > > > > > That is because connectors have both a big variety of
> > scenarios
> > > > > they
> > > > > > > get
> > > > > > > > > used in (only through use and many small contributions do
> > they
> > > > > become
> > > > > > > > > really useful over time) and at the same time, and
> committers
> > > do
> > > > > not
> > > > > > > use
> > > > > > > > > the connector themselves and usually cannot foresee too
> well
> > > what
> > > > > is
> > > > > > > > > needed.
> > > > > > > > >
> > > > > > > > > Further more, a missing connector (or connector feature) is
> > > > often a
> > > > > > > > bigger
> > > > > > > > > show stopper for users than a missing API or system
> feature.
> > > > > > > > >
> > > > > > > > > Along these lines of thougt, the conclusion would be to
> take
> > > the
> > > > > > Pulsar
> > > > > > > > > connector now, focus the review on legal/dependencies/rough
> > > code
> > > > > > style
> > > > > > > > and
> > > > > > > > > conventions, label it as "beta" (in the sense of "new code"
> > > that
> > > > is
> > > > > > > "not
> > > > > > > > > yet tested through longer use") and go ahead. And then
> evolve
> > > it
> > > > > > > quickly
> > > > > > > > > without putting formal blockers in the way, meaning also
> > > adding a
> > > > > new
> > > > > > > > FLIP
> > > > > > > > > 27 version when it is there.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yijie,
> > > > > > > > > >
> > > > > > > > > > Could you please follow the FLIP process to start a new
> > FLIP
> > > > > > > > [DISCUSSION]
> > > > > > > > > > thread in the mailing list?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > > > > > >
> > > > > > > > > > I see two FLIP-69 discussion in the mailing list now. So
> > > there
> > > > > is a
> > > > > > > > FLIP
> > > > > > > > > > number collision. Can you change the FLIP number to 72?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yijie,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > > > > > Would you mind enabling comments/suggestions on the
> > google
> > > > doc
> > > > > > > link?
> > > > > > > > > This
> > > > > > > > > > > way the contributors from the community can comment on
> > the
> > > > doc.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Rong
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > I've drafted a FLIP that describes the current design
> > of
> > > > the
> > > > > > > Pulsar
> > > > > > > > > > > > connector:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > > > > > >
> > > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Yijie
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Sorry for joining the discussion late and thanks
> > Yijie
> > > &
> > > > > > Sijie
> > > > > > > > for
> > > > > > > > > > > > driving
> > > > > > > > > > > > > the discussion.
> > > > > > > > > > > > > I also think the Pulsar connector would be a very
> > > > valuable
> > > > > > > > addition
> > > > > > > > > > to
> > > > > > > > > > > > > Flink. I can also help out a bit on the review side
> > :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding the timeline, I also share concerns with
> > > Becket
> > > > > on
> > > > > > > the
> > > > > > > > > > > > > relationship between the new Pulsar connector and
> > > > FLIP-27.
> > > > > > > > > > > > > There's also another discussion just started by
> > Stephan
> > > > on
> > > > > > > > dropping
> > > > > > > > > > > Kafka
> > > > > > > > > > > > > 9/10 support on next Flink release [1].  Although
> the
> > > > > > situation
> > > > > > > > is
> > > > > > > > > > > > somewhat
> > > > > > > > > > > > > different, and Kafka 9/10 connector has been in
> Flink
> > > for
> > > > > > > almost
> > > > > > > > > 3-4
> > > > > > > > > > > > years,
> > > > > > > > > > > > > based on the discussion I am not sure if a major
> > > version
> > > > > > > release
> > > > > > > > > is a
> > > > > > > > > > > > > requirement for removing old connector supports.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think there shouldn't be a blocker if we agree
> the
> > > old
> > > > > > > > connector
> > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is
> there.
> > > As
> > > > > > > Stephan
> > > > > > > > > > > stated,
> > > > > > > > > > > > it
> > > > > > > > > > > > > is easier to contribute the source sooner and
> adjust
> > it
> > > > > > later.
> > > > > > > > > > > > > We should also ensure we clearly communicate the
> > > message:
> > > > > for
> > > > > > > > > > example,
> > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27
> > > connector
> > > > > page
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Rong
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Technically speaking, removing the old connector
> > code
> > > > is
> > > > > a
> > > > > > > > > > backwards
> > > > > > > > > > > > > > incompatible change which requires a major
> version
> > > > bump,
> > > > > > i.e.
> > > > > > > > > Flink
> > > > > > > > > > > > 2.x.
> > > > > > > > > > > > > > Given that we don't have a clear plan on when to
> > have
> > > > the
> > > > > > > next
> > > > > > > > > > major
> > > > > > > > > > > > > > version release, it seems unclear how long the
> old
> > > > > > connector
> > > > > > > > code
> > > > > > > > > > > will
> > > > > > > > > > > > be
> > > > > > > > > > > > > > there if we check it in right now. Or will we
> > remove
> > > > the
> > > > > > old
> > > > > > > > > > > connector
> > > > > > > > > > > > > > without a major version bump? In any case, it
> > sounds
> > > > not
> > > > > > > quite
> > > > > > > > > user
> > > > > > > > > > > > > > friendly to the those who might use the old
> Pulsar
> > > > > > > connector. I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > > sure
> > > > > > > > > > > > > > if it is worth these potential problems in order
> to
> > > > have
> > > > > > the
> > > > > > > > > Pulsar
> > > > > > > > > > > > source
> > > > > > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Agreed, if we check in the old code, we should
> > make
> > > > it
> > > > > > > clear
> > > > > > > > > that
> > > > > > > > > > > it
> > > > > > > > > > > > will
> > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based version
> > of
> > > > the
> > > > > > > > > connector
> > > > > > > > > > is
> > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > We should not commit to maintaining the old
> > > version,
> > > > > that
> > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > indeed
> > > > > > > > > > > > > > > too much overhead.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Yes, the overhead would just be review
> > capacity.
> > > In
> > > > > > > fact, I
> > > > > > > > > am
> > > > > > > > > > > not
> > > > > > > > > > > > > > > worrying
> > > > > > > > > > > > > > > > too much about the review capacity. That is
> > just
> > > a
> > > > > one
> > > > > > > time
> > > > > > > > > > cost.
> > > > > > > > > > > > My
> > > > > > > > > > > > > > > > concern is mainly about the long term burden.
> > > > Assume
> > > > > we
> > > > > > > > have
> > > > > > > > > > new
> > > > > > > > > > > > source
> > > > > > > > > > > > > > > > interface ready in 1.10 with newly added
> Pulsar
> > > > > > > connectors
> > > > > > > > in
> > > > > > > > > > old
> > > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to
> new
> > > > > source
> > > > > > > > > > interface,
> > > > > > > > > > > > the
> > > > > > > > > > > > > > old
> > > > > > > > > > > > > > > > Pulsar connector might be deprecated almost
> > > > > immediately
> > > > > > > > after
> > > > > > > > > > > > checked
> > > > > > > > > > > > > > in,
> > > > > > > > > > > > > > > > but we may still have to maintain two code
> > bases.
> > > > For
> > > > > > the
> > > > > > > > > > > existing
> > > > > > > > > > > > > > > > connectors, we have to do that anyways. But
> it
> > > > would
> > > > > be
> > > > > > > > good
> > > > > > > > > to
> > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > introducing a new connector with the same
> > > problem.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan Ewen
> <
> > > > > > > > > [hidden email]
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Nice to see this lively discussion about
> the
> > > > Pulsar
> > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a
> > > community
> > > > > > > package
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Looks like the discussion is more going
> > towards
> > > > > > > > > > contribution. I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > is good, especially if we think that we
> want
> > to
> > > > > > build a
> > > > > > > > > > > similarly
> > > > > > > > > > > > > > deep
> > > > > > > > > > > > > > > > > integration with Pulsar as we have for
> > example
> > > > with
> > > > > > > > Kafka.
> > > > > > > > > > The
> > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > already looks like a more thorough
> connector
> > > than
> > > > > > many
> > > > > > > > > others
> > > > > > > > > > > we
> > > > > > > > > > > > have
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the repository.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > With either a repo split, or the new build
> > > > system,
> > > > > I
> > > > > > > hope
> > > > > > > > > > that
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Becket offered some help already, I can
> also
> > > > help a
> > > > > > > bit.
> > > > > > > > I
> > > > > > > > > > hope
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually making
> > some
> > > > > > PoC-ing
> > > > > > > > > > > > progress, but
> > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > take 2 more months, I would estimate,
> before
> > it
> > > > is
> > > > > > > fully
> > > > > > > > > > > > available.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > If we want to be on the safe side with the
> > > > > > > contribution,
> > > > > > > > we
> > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > contribute the source sooner and adjust it
> > > later.
> > > > > > That
> > > > > > > > > would
> > > > > > > > > > > also
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > in case things get crazy towards the 1.10
> > > feature
> > > > > > > freeze
> > > > > > > > > and
> > > > > > > > > > it
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > hard to find time to review the new
> changes.
> > > > > > > > > > > > > > > > > What would be the overhead of contributing
> > now?
> > > > > Given
> > > > > > > > that
> > > > > > > > > > the
> > > > > > > > > > > > code
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > already there, it looks like it would be
> only
> > > > > review
> > > > > > > > > > capacity,
> > > > > > > > > > > > right?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie
> Shen <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for your attention and the
> promotion
> > > of
> > > > > this
> > > > > > > > work.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as
> possible
> > > for
> > > > > more
> > > > > > > > > > specific
> > > > > > > > > > > > > > > > discussions.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not
> > > reached
> > > > a
> > > > > > > > > consensus.
> > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > I will explain all the functionalities of
> > the
> > > > > > > existing
> > > > > > > > > > > > connector in
> > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and
> > > Catalog)
> > > > to
> > > > > > > > > continue
> > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket
> Qin
> > <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > If we agree that the goal is to have
> > Pulsar
> > > > > > > connector
> > > > > > > > > in
> > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar connector
> > to
> > > > > Flink
> > > > > > > main
> > > > > > > > > > repo
> > > > > > > > > > > > as it
> > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink
> right
> > > away
> > > > > as
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > > no
> > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. Flink
> > 1.10
> > > > is
> > > > > > > going
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > code
> > > > > > > > > > > > > > > > freeze
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > late Nov and let's say we give a month
> to
> > > the
> > > > > > > > > development
> > > > > > > > > > > and
> > > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have
> FLIP-27
> > > by
> > > > > late
> > > > > > > > Oct.
> > > > > > > > > > > There
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > > > > > weeks. Personally I think it is doable.
> > If
> > > > > > FLIP-27
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > ready
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > late
> > > > > > > > > > > > > > > > > > > Oct, we can review and check in Pulsar
> > > > > connector
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > interface. This means we will have
> Pulsar
> > > > > > connector
> > > > > > > > in
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > either
> > > > > > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar
> sink
> > > and
> > > > > > source
> > > > > > > > > > checked
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > > > > > it might make sense to have two FLIPs,
> > one
> > > > for
> > > > > > > Pulsar
> > > > > > > > > > sink
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > Pulsar source. And we can start the
> work
> > on
> > > > > > Pulsar
> > > > > > > > sink
> > > > > > > > > > > right
> > > > > > > > > > > > > > away.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie
> Guo
> > <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > What's the take from Flink community?
> > > Shall
> > > > > we
> > > > > > > wait
> > > > > > > > > for
> > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > > > > > proceed to next steps? And what the
> > next
> > > > > steps
> > > > > > > are?
> > > > > > > > > :-)
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM Bowen
> > Li <
> > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector
> in
> > > > Flink
> > > > > > can
> > > > > > > > be a
> > > > > > > > > > > good
> > > > > > > > > > > > > > mutual
> > > > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Another perspective is that Pulsar
> > > > > connector
> > > > > > is
> > > > > > > > the
> > > > > > > > > > 1st
> > > > > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > that integrates with Flink's
> metadata
> > > > > > > management
> > > > > > > > > > system
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the
> > > integration
> > > > > > turns
> > > > > > > > out
> > > > > > > > > > and
> > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, which
> > are
> > > > > > > currently
> > > > > > > > in
> > > > > > > > > > > > Beta, to
> > > > > > > > > > > > > > > > cater
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm in
> > > favor
> > > > of
> > > > > > > > merging
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller sized
> > > PRs,
> > > > > e.g.
> > > > > > > > maybe
> > > > > > > > > > one
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > > > > > source/sink functionalities and
> > another
> > > > for
> > > > > > > > schema
> > > > > > > > > > and
> > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > > > > > just to make them easier to review.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait for
> > > > > FLIP-27.
> > > > > > > But
> > > > > > > > I
> > > > > > > > > > > don't
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > > > > > should be a blocker in cases where
> it
> > > > > cannot
> > > > > > > make
> > > > > > > > > its
> > > > > > > > > > > way
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount of
> > time
> > > > for
> > > > > > > > > > committers
> > > > > > > > > > > to
> > > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt to
> > new
> > > > > > > > interfaces.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM
> Becket
> > > > Qin <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > You are right. It all depends on
> > when
> > > > the
> > > > > > new
> > > > > > > > > > source
> > > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it
> > would
> > > > be
> > > > > > > there
> > > > > > > > in
> > > > > > > > > > > > about a
> > > > > > > > > > > > > > > month
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It
> would
> > > > also
> > > > > be
> > > > > > > > good
> > > > > > > > > to
> > > > > > > > > > > > hear
> > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > > > > > and Stephan think as they are
> also
> > > > > involved
> > > > > > > in
> > > > > > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > In general I think we should have
> > > > Pulsar
> > > > > > > > > connector
> > > > > > > > > > in
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > > > > > preferably with the new source
> > > > interface.
> > > > > > We
> > > > > > > > can
> > > > > > > > > > also
> > > > > > > > > > > > check
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > > > > > with old source interface, but I
> > > > suspect
> > > > > > few
> > > > > > > > > users
> > > > > > > > > > > > will use
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > next official release. Therefore,
> > it
> > > > > seems
> > > > > > > > > > reasonable
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > wait a
> > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > to see whether we can jump to the
> > new
> > > > > > source
> > > > > > > > > > > > interface. As
> > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, waiting a
> > > > little
> > > > > > bit
> > > > > > > > > > doesn't
> > > > > > > > > > > > seem
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > hurt
> > > > > > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM
> Till
> > > > > > Rohrmann
> > > > > > > <
> > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the problem
> > > would
> > > > be
> > > > > > if
> > > > > > > we
> > > > > > > > > > > > committed
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > connector before the new source
> > > > > interface
> > > > > > > is
> > > > > > > > > > ready.
> > > > > > > > > > > > If I
> > > > > > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to
> > support
> > > > the
> > > > > > old
> > > > > > > > > source
> > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > existing connectors. By
> checking
> > it
> > > > in
> > > > > > > early
> > > > > > > > I
> > > > > > > > > > > could
> > > > > > > > > > > > see
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > our users could start using the
> > > > > connector
> > > > > > > > > > earlier.
> > > > > > > > > > > > > > > Moreover,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar
> > integration
> > > > is
> > > > > > > being
> > > > > > > > > > > delayed
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > > > > source interface should be
> > delayed.
> > > > The
> > > > > > > only
> > > > > > > > > > > > downside I
> > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > > > > > review effort and potential
> fixes
> > > > which
> > > > > > > might
> > > > > > > > > be
> > > > > > > > > > > > > > irrelevant
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > source interface
> implementation.
> > I
> > > > > guess
> > > > > > it
> > > > > > > > > > mainly
> > > > > > > > > > > > > > depends
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > > > > > we are when the new source
> > > interface
> > > > > will
> > > > > > > be
> > > > > > > > > > ready.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56 AM
> > > Becket
> > > > > > Qin <
> > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your
> > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Just want to have some update
> > on
> > > > > > FLIP-27.
> > > > > > > > > > > Although
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > discussion thread has been
> > quiet
> > > > for
> > > > > > some
> > > > > > > > > > time, a
> > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink
> community
> > > > were
> > > > > > > > actually
> > > > > > > > > > > > > > prototyping
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > > > > > We have made some good
> progress
> > > > there
> > > > > > but
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > > update
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > > > > > after the entire thing is
> > > verified
> > > > to
> > > > > > > work
> > > > > > > > in
> > > > > > > > > > > case
> > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the
> > > > > implementation.
> > > > > > I
> > > > > > > > > don't
> > > > > > > > > > > > have an
> > > > > > > > > > > > > > > > exact
> > > > > > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be
> within
> > a
> > > > > month
> > > > > > or
> > > > > > > > so.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the
> > current
> > > > > Flink
> > > > > > > > Pulsar
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be
> > good
> > > to
> > > > > > avoid
> > > > > > > > the
> > > > > > > > > > > case
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some
> > review
> > > > > > efforts
> > > > > > > > and
> > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39
> AM
> > > > Yijie
> > > > > > > Shen <
> > > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback
> > and
> > > > > > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal of
> > the
> > > > > > > connector
> > > > > > > > > has
> > > > > > > > > > > > always
> > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > > users with the latest
> > features
> > > of
> > > > > > both
> > > > > > > > > > systems
> > > > > > > > > > > as
> > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the
> > > > connector
> > > > > > to
> > > > > > > > > Flink
> > > > > > > > > > > and
> > > > > > > > > > > > hope
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback
> from
> > > > Flink
> > > > > > > > experts
> > > > > > > > > > to
> > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed its
> > > > > existence
> > > > > > > at
> > > > > > > > > the
> > > > > > > > > > > > > > beginning
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > > > > > the connector
> implementation
> > > > based
> > > > > on
> > > > > > > > Flink
> > > > > > > > > > > 1.9;
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > build a connector that
> > supports
> > > > > both
> > > > > > > > batch
> > > > > > > > > > and
> > > > > > > > > > > > stream
> > > > > > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > > > > > However, it has been
> inactive
> > > for
> > > > > > some
> > > > > > > > > time,
> > > > > > > > > > so
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > > decided
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > > a connector with most of
> the
> > > new
> > > > > > > > features,
> > > > > > > > > > such
> > > > > > > > > > > > as
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API
> > first.
> > > We
> > > > > > will
> > > > > > > > pay
> > > > > > > > > > > > attention
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually and
> > > > > > incorporate
> > > > > > > it
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status
> of
> > > the
> > > > > > > > connector,
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> > > > > repository
> > > > > > > and
> > > > > > > > > > aimed
> > > > > > > > > > > to
> > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are
> > also
> > > > > happy
> > > > > > to
> > > > > > > > > hear
> > > > > > > > > > > > > > > suggestions
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > supervision from the Flink
> > > > > community
> > > > > > to
> > > > > > > > > > improve
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > > > > > performance of the
> connector
> > > > > > > > continuously.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 5:59
> > AM
> > > > > Sijie
> > > > > > > Guo
> > > > > > > > <
> > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the
> > > > comments
> > > > > > and
> > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the
> > main
> > > > > > question
> > > > > > > > > here
> > > > > > > > > > is
> > > > > > > > > > > > > > about -
> > > > > > > > > > > > > > > > > "how
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > community maintain the
> > > > > connector?".
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts
> from
> > > > > myself.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and where
> to
> > > > host
> > > > > > this
> > > > > > > > > > > > integration
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > kind
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there can
> > be
> > > > many
> > > > > > > ways
> > > > > > > > to
> > > > > > > > > > > > achieve
> > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > > > As part of the
> > contribution,
> > > > what
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > looking
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > here
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > > > > > communities can build the
> > > > > > > collaboration
> > > > > > > > > > > > > > relationship
> > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > > > > > the integration between
> > > Pulsar
> > > > > and
> > > > > > > > Flink.
> > > > > > > > > > > Even
> > > > > > > > > > > > we
> > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in
> Flink
> > > > > > > community.
> > > > > > > > We
> > > > > > > > > > are
> > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we
> > have
> > > > less
> > > > > > > > > > experiences
> > > > > > > > > > > > in
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > community. In order to
> make
> > > > sure
> > > > > we
> > > > > > > > > > maintain
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality
> pulsar-flink
> > > > > > > integration
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > who
> > > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need
> some
> > > help
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > experts
> > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been following
> > > > FLIP-27
> > > > > > > for a
> > > > > > > > > > > while.
> > > > > > > > > > > > > > > > Originally
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the
> > > connectors
> > > > > back
> > > > > > > > after
> > > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in
> > > FLIP-27.
> > > > > But
> > > > > > we
> > > > > > > > > > decided
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > initiate
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > > > > > as early as possible.
> > Because
> > > > we
> > > > > > > > believe
> > > > > > > > > > > there
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than later.
> > As
> > > > part
> > > > > > of
> > > > > > > > > > > > contribution,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > community understand more
> > > about
> > > > > > > Pulsar
> > > > > > > > > and
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help
> Flink
> > > > > > community
> > > > > > > > > > verify
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g.
> > > catalog
> > > > > > API).
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> 5:24
> > > AM
> > > > > > Becket
> > > > > > > > > Qin <
> > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the interest
> > in
> > > > > > > > contributing
> > > > > > > > > > the
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think
> > having
> > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to
> > Flink.
> > > > So
> > > > > I
> > > > > > am
> > > > > > > > > happy
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would also
> > > like
> > > > to
> > > > > > > > provide
> > > > > > > > > > > some
> > > > > > > > > > > > > > > context
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors
> > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink
> > > > maintains
> > > > > > its
> > > > > > > > > > > > connector has
> > > > > > > > > > > > > > > hit
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more
> > > connectors
> > > > > > > coming
> > > > > > > > > into
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > repo,
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long
> > build
> > > > and
> > > > > > > > testing
> > > > > > > > > > > > time. To
> > > > > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do
> the
> > > > > > following:
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the
> > connectors
> > > > > into
> > > > > > a
> > > > > > > > > > separate
> > > > > > > > > > > > > > > > repository.
> > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to
> potential
> > > > > solution
> > > > > > > to
> > > > > > > > > > > shorten
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the
> > connectors
> > > > to
> > > > > > stay
> > > > > > > > as
> > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good support
> > for
> > > > > > > > > functionality
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create a
> > > Flink
> > > > > > > > Ecosystem
> > > > > > > > > > > > project
> > > > > > > > > > > > > > > > website
> > > > > > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > > > > > through some final
> > approval
> > > > > > > process.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above
> efforts,
> > it
> > > > > would
> > > > > > > be
> > > > > > > > > > great
> > > > > > > > > > > to
> > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as an
> > > > > ecosystem
> > > > > > > > > project
> > > > > > > > > > > with
> > > > > > > > > > > > > > great
> > > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the
> > Flink
> > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > tested
> > > > > > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn something
> to
> > > > > > maintain
> > > > > > > it
> > > > > > > > > as
> > > > > > > > > > an
> > > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test
> > coverage.
> > > If
> > > > > the
> > > > > > > > > quality
> > > > > > > > > > > as
> > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may as
> > > well
> > > > > > adopt
> > > > > > > it
> > > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > main
> > > > > > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing
> > effort
> > > > is
> > > > > > > > FLIP-27
> > > > > > > > > > > where
> > > > > > > > > > > > we
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source connector
> > > > > > architecture
> > > > > > > > and
> > > > > > > > > > > > > > interface.
> > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. Therefore
> > > > timing
> > > > > > > wise,
> > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > > > are
> > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main
> repo, I
> > > am
> > > > > > > > wondering
> > > > > > > > > if
> > > > > > > > > > > we
> > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar
> connector
> > > > adapt
> > > > > to
> > > > > > > the
> > > > > > > > > new
> > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> > 4:32
> > > > PM
> > > > > > > > Chesnay
> > > > > > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried
> that
> > we
> > > > may
> > > > > > end
> > > > > > > > up
> > > > > > > > > > > > repeating
> > > > > > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already 2
> > > > attempts
> > > > > > at
> > > > > > > > > > > > contributing a
> > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed
> because
> > > no
> > > > > > > > committer
> > > > > > > > > > was
> > > > > > > > > > > > > > getting
> > > > > > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor opening a
> > > > > dedicated
> > > > > > > > > > > discussion
> > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and
> getting
> > > > > several
> > > > > > > +1's
> > > > > > > > > > from
> > > > > > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really make
> > > sure
> > > > > that
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it will
> > > > actually
> > > > > > get
> > > > > > > > the
> > > > > > > > > > > > attention
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm inclined
> > to
> > > > > > > recommend
> > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could link
> to
> > > it
> > > > > from
> > > > > > > the
> > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming
> page
> > > for
> > > > > > > sharing
> > > > > > > > > > > > artifacts
> > > > > > > > > > > > > > > among
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that
> > > anyway?),
> > > > > > this
> > > > > > > > may
> > > > > > > > > > be a
> > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 10:16,
> > Till
> > > > > > > Rohrmann
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for
> > > starting
> > > > > > this
> > > > > > > > > > > discussion
> > > > > > > > > > > > > > > Yijie.
> > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would be
> a
> > > very
> > > > > > > > valuable
> > > > > > > > > > > > addition
> > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and it
> > > would
> > > > > > > further
> > > > > > > > > > > expand
> > > > > > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project
> > > > perspective
> > > > > it
> > > > > > > > makes
> > > > > > > > > > > > sense for
> > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream
> > project.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> > > concern/question
> > > > is
> > > > > > how
> > > > > > > > can
> > > > > > > > > > the
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We have
> > seen
> > > > in
> > > > > > the
> > > > > > > > past
> > > > > > > > > > > that
> > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively developed
> > > > > components
> > > > > > > > > because
> > > > > > > > > > > > they
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > external system and
> > > with
> > > > > > Flink.
> > > > > > > > > Given
> > > > > > > > > > > > that
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
> > > maintaining,
> > > > > > > > improving
> > > > > > > > > > and
> > > > > > > > > > > > > > > evolving
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that we
> > can
> > > > > > achieve
> > > > > > > > > this.
> > > > > > > > > > > > Hence,
> > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019
> > at
> > > > 2:03
> > > > > > AM
> > > > > > > > > Sijie
> > > > > > > > > > > Guo
> > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the
> main
> > > > > driver
> > > > > > > > behind
> > > > > > > > > > > > > > FLINK-9641
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context
> on
> > > > this.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and
> > > > FLINK-9168
> > > > > > was
> > > > > > > > > > created
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink.
> The
> > > > > > > integration
> > > > > > > > > was
> > > > > > > > > > > done
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a
> > year
> > > > ago
> > > > > > and
> > > > > > > we
> > > > > > > > > > ended
> > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for Pulsar
> > > users
> > > > to
> > > > > > use
> > > > > > > > > Flink
> > > > > > > > > > > to
> > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration is
> > > > pretty
> > > > > > > simple
> > > > > > > > > and
> > > > > > > > > > > > there
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past year,
> we
> > > > have
> > > > > > > made a
> > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as
> the
> > > > > > > first-class
> > > > > > > > > > > > citizen in
> > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing
> > > engines
> > > > > for
> > > > > > > > > > processing
> > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to
> rethink
> > > how
> > > > > to
> > > > > > > > > > integrate
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> > > > > pulsar-flink
> > > > > > > > > > > connectors
> > > > > > > > > > > > from
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API
> and
> > > > > catalog
> > > > > > > API
> > > > > > > > as
> > > > > > > > > > the
> > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. With
> > that
> > > > > being
> > > > > > > > said,
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register
> > > pulsar
> > > > > as a
> > > > > > > > flink
> > > > > > > > > > > > catalog
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using
> Flink
> > > SQL.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an example
> > > about
> > > > > how
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also
> > > written a
> > > > > > blog
> > > > > > > > post
> > > > > > > > > > > > > > explaining
> > > > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with
> Flink
> > > 1.9
> > > > > and
> > > > > > > > what
> > > > > > > > > > are
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe Pulsar
> is
> > > not
> > > > > > just
> > > > > > > a
> > > > > > > > > > simple
> > > > > > > > > > > > data
> > > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be a
> > > fully
> > > > > > > > integrated
> > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink,
> source,
> > > > > > > > > schema/catalog
> > > > > > > > > > > and
> > > > > > > > > > > > > > > state).
> > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can
> > create
> > > a
> > > > > > great
> > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first,
> > > unified
> > > > > > data
> > > > > > > > > > > > processing.
> > > > > > > > > > > > > > > Since
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute Pulsar
> > > > > > integration
> > > > > > > to
> > > > > > > > > > Flink
> > > > > > > > > > > > here,
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, improve
> > and
> > > > > evolve
> > > > > > > the
> > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both Flink
> > and
> > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give
> you a
> > > bit
> > > > > > more
> > > > > > > > > > > background
> > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let
> me
> > > know
> > > > > > what
> > > > > > > > are
> > > > > > > > > > your
> > > > > > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3,
> 2019
> > at
> > > > > 11:54
> > > > > > > AM
> > > > > > > > > Yun
> > > > > > > > > > > > Tang <
> > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that
> > Pulsar
> > > > > > becomes
> > > > > > > > > more
> > > > > > > > > > > and
> > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more
> people
> > > > > willing
> > > > > > to
> > > > > > > > > > > > contribute to
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any
> further
> > > > > > > discussion,
> > > > > > > > > > would
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > please
> > > > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the relationship
> > > > between
> > > > > > this
> > > > > > > > > > thread
> > > > > > > > > > > to
> > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and
> sink
> > > [2]
> > > > > > > > > connector?
> > > > > > > > > > > > Will the
> > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or
> > totally
> > > > > > > different
> > > > > > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > > > > >
> > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie Shen
> <
> > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday,
> > > > September
> > > > > 3,
> > > > > > > > 2019
> > > > > > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> > > > [hidden email]
> > > > > <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject:
> [DISCUSS]
> > > > > > Contribute
> > > > > > > > > > Pulsar
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink
> > Community!
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to
> > open
> > > > the
> > > > > > > > > discussion
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0]
> back
> > to
> > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
> > > introduction
> > > > > to
> > > > > > > > Apache
> > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache Pulsar[1]
> > is a
> > > > > > > > > multi-tenant,
> > > > > > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub messaging
> > > > system.
> > > > > > > > Pulsar
> > > > > > > > > > > > includes
> > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support
> for
> > > > > multiple
> > > > > > > > > > clusters
> > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> > > > geo-replication
> > > > > of
> > > > > > > > > > messages
> > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end
> > > latency,
> > > > > > > > seamless
> > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed
> > > message
> > > > > > > delivery
> > > > > > > > > > with
> > > > > > > > > > > > > > > persistent
> > > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by
> Apache
> > > > > > > BookKeeper.
> > > > > > > > > > > > Nowadays,
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> > > > > companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status of
> > > Pulsar
> > > > > > Flink
> > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar Flink
> > > > > connector
> > > > > > we
> > > > > > > > are
> > > > > > > > > > > > planning
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and
> > > Pulsar
> > > > > > 2.4.0.
> > > > > > > > The
> > > > > > > > > > > main
> > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a
> > > streaming
> > > > > > > source
> > > > > > > > > with
> > > > > > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink streaming
> > > > results
> > > > > to
> > > > > > > > > Pulsar
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update this
> > to
> > > > > > > > exactly-once
> > > > > > > > > > as
> > > > > > > > > > > > well
> > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction
> > features
> > > > > ready
> > > > > > in
> > > > > > > > its
> > > > > > > > > > > 2.5.0
> > > > > > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon
> Flink
> > > new
> > > > > > Table
> > > > > > > > API
> > > > > > > > > > Type
> > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> > > > > (de)serialize
> > > > > > > > > > messages
> > > > > > > > > > > > with
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate with
> > > Flink
> > > > > new
> > > > > > > > > Catalog
> > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar
> > topics
> > > as
> > > > > > > tables
> > > > > > > > in
> > > > > > > > > > > > Table API
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > > > > > https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[Becket Qin]

Thanks, Stephan.

Sounds good to me. We can still try our best to get new Pulsar connector in
Flink 1.10. In case we do not have time to do that, we will prominently
link the Pulsar connector from the Flink connector docs.

Thanks,

Jiangjie (Becket) Qin

On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <[hidden email]> wrote:

> Okay, I see your point, Becket.
>
> Then let us prominently link the Pulsar connector from the Flink connector
> docs then, so that users can find it easily.
>
> As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute a
> new connector.
>
> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]> wrote:
>
> > Hi Stephan,
> >
> > I have no doubt about the value of adding Pulsar connector to Flink repo.
> > My concern is about how exactly we are going to do it.
> >
> > As mentioned before, I believe that we can handle connectors more
> > > pragmatically and less strict than the core of Flink, if it helps
> > unlocking
> > > users.
> >
> > I can see the benefit of being less restrict for the initial connector
> code
> > adoption. However, I don't think we should be less restrict on the
> > maintenance commitment once the code is in Flink repo. It only makes
> sense
> > to check in something and ask users to use if we plan to maintain it.
> >
> > If I understand correctly, the current plan so far is following:
> > 1. release 1.10
> >    - Check in Pulsar connector on old interface and label it as beta
> > version.
> >    - encourage users to try it and report bugs.
> > 2. release 1.11
> >    - Check in Pulsar connector on new interface (a.k.a new Pulsar
> > connector) and label it as beta version
> >    - Deprecate the old Pulsar connector
> >    - Fix bugs reported on old Pulsar connector from release 1.10
> >    - Ask users to migrate from old Pulsar connector to new Pulsar
> connector
> > 3. release 1.12
> >    - Announce end of support for old Pulsar connector and remove the code
> >    - Fix bugs reported on new Pulsar connector.
> >
> > If this is the plan, it seems neither Flink nor the users trying the old
> > Pulsar connector will benefit from this experimental old Pulsar
> connector,
> > because whatever feedbacks we got or bugs we fix on the old Pulsar
> > connector are immediately thrown away in one or two releases.
> >
> > If we check in the old Pulsar connector right now, the only option I see
> is
> > to maintain it for a while (e.g. a year or more). IMO, the immediate
> > deprecation and code removal hurts the users much more than asking them
> to
> > wait for another release. I personally think that we can avoid this
> > maintenance burden by going directly to the new Pulsar connector,
> > especially given that users can still use the connector even if they are
> > not in Flink repo. That said, I am OK with maintaining both old and new
> > Pulsar connector if we believe that having the Pulsar connector available
> > right now in Flink repo is more important.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]> wrote:
> >
> > > My assumption is as Sijie's, that once the connector is either part of
> > > Flink, or part of the streamnative repo. No double maintenance.
> > >
> > > I feel this discussion is very much caught in problems that are all
> > > solvable if we want to solve them.
> > > Maybe we can think what our goal for users and the communities is?
> > >
> > >   - Do we want to help build a relationship between the Pulsar and
> Flink
> > > open source communities?
> > >   - Will users find a connector in the streamnative repository?
> > >   - Will users trust a connector that is not part of Flink as much?
> > >
> > > And then decide what is best according to the overall goals there.
> > > As mentioned before, I believe that we can handle connectors more
> > > pragmatically and less strict than the core of Flink, if it helps
> > unlocking
> > > users.
> > >
> > > Best,
> > > Stephan
> > >
> > >
> > >
> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]> wrote:
> > >
> > > > Thanks Becket.
> > > >
> > > > I think it is better for the Flink community to judge the benefits of
> > > doing
> > > > this. I was trying to provide some views from outsiders.
> > > >
> > > > Thanks,
> > > > Sijie
> > > >
> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi Sijie,
> > > > >
> > > > > Yes, we will have to support existing old connectors and new
> > connectors
> > > > in
> > > > > parallel for a while. We have to take that maintenance overhead
> > because
> > > > > existing connectors have been used by the users for a long time. I
> > > guess
> > > > It
> > > > > may take at least a year for us to fully remove the old connectors.
> > > > >
> > > > > Process wise, we can do the same for Pulsar connector. But I am not
> > > sure
> > > > if
> > > > > we want to have the same burden on Pulsar connector, and I would
> like
> > > to
> > > > > understand the benefit of doing that.
> > > > >
> > > > > For users, the benefit of having the old Pulsar connector checked
> in
> > > > seems
> > > > > limited because 1) that code base will be immediately deprecated in
> > the
> > > > > next release in 3-4 months; 2) users can always use it even if it
> is
> > > not
> > > > in
> > > > > the Flink code base. Admittedly it is not as convenient as having
> it
> > in
> > > > > Flink code base, but doesn't seem super either. And after 3-4
> months,
> > > > users
> > > > > can just use the new connector in Flink repo.
> > > > >
> > > > > For Flink developers, the old connector code base is not something
> > that
> > > > we
> > > > > want to evolve later. Instead, these code will be deprecated and
> > > > > removed. So why do we want to get a beta version out to attract
> > people
> > > to
> > > > > use something we don't want to maintain?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]>
> > wrote:
> > > > >
> > > > > > Thanks everyone here. Sorry for jumping into the discussion here.
> > > > > >
> > > > > > I am not very familiar about the deprecation process in Flink.
> If I
> > > > > > misunderstood the process, please fix me.
> > > > > >
> > > > > > As far as I understand, FLIP-27 is introducing a new unified API
> > for
> > > > > > connectors. After it introduces the new API
> > > > > > and before moving all the existing connectors from old API to new
> > > API,
> > > > > both
> > > > > > old ApI and new API will co-exist
> > > > > > for a while until Flink moves all existing connectors to new API.
> > So
> > > > the
> > > > > > Pulsar connector (using old API) can
> > > > > > follow the deprecation process with other connector using old API
> > and
> > > > the
> > > > > > deprecation of old API, no?
> > > > > >
> > > > > > If that's the case, I think contributing the current connector
> back
> > > to
> > > > > > Flink rather than maintaining it outside Flink
> > > > > > would provide a bit more benefits. We can deprecate the existing
> > > > > > streamnative/pulsar-flink repo and point the users
> > > > > > to use the connector in Flink repo. So all the review processes
> > will
> > > > > happen
> > > > > > within Flink for both old connector and
> > > > > > new connector. It also reduces the confusions for the users as
> the
> > > > > > documentation and code base happen in one place.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > - Sijie
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks for the explanation, Stephan. I have a few questions /
> > > > thoughts.
> > > > > > >
> > > > > > > So that means we will remove the old connector without a major
> > > > version
> > > > > > > bump, is that correct?
> > > > > > >
> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11
> connectors
> > > > will
> > > > > > > always work because we saw some dependency class collisions in
> > the
> > > > > past.
> > > > > > To
> > > > > > > make it safe we may have to maintain the old code for one more
> > > > release.
> > > > > > >
> > > > > > > To be honest I am still wondering if we have to put the old
> > > connector
> > > > > in
> > > > > > > Flink repo. if we check in the old connector to Flink. We will
> > end
> > > up
> > > > > in
> > > > > > > the following situation:
> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > > > > 2. Old connector in Flink Repo, which may be different from the
> > one
> > > > in
> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in
> 1.12)
> > > > > > > 3. New connector in Flink Repo.
> > > > > > >
> > > > > > > We need to think about how to make the users in each case
> happy.
> > > > > > > - For users of (1), I assume Sijie and Yijie will have to
> > maintain
> > > > the
> > > > > > code
> > > > > > > a bit longer for its own compatibility even after we have (2).
> In
> > > > that
> > > > > > > case, bugs found in old connector may or may not need to be
> fixed
> > > in
> > > > > both
> > > > > > > Flink and the streamnative/pulsar-flink repo.
> > > > > > > - For users of (2), will we provide bug fixes? If we do, it
> will
> > > be a
> > > > > > > little awkward because those bug fixes will be immediately
> > > deprecated
> > > > > in
> > > > > > > 1.11, and removed in 1.12. So we are essentially asking users
> to
> > > > > migrate
> > > > > > > away from the bug fix. After Flink 1.12, users may still have
> to
> > > > switch
> > > > > > to
> > > > > > > use (3) due to the potential dependency class conflicts
> mentioned
> > > > > above.
> > > > > > > - Users of (3) have a much easier life and don't need to worry
> > too
> > > > > much.
> > > > > > >
> > > > > > > The above story seems a little complicated to tell. I think it
> > will
> > > > be
> > > > > > much
> > > > > > > easier to not have (2) at all.
> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > > > > > 3. New connector in Flink Repo.
> > > > > > >
> > > > > > > - Old connector will only be maintained in
> > > streamnative/pulsar-flink
> > > > > repo
> > > > > > > until it is fully deprecated. Users can always use the existing
> > > > Pulsar
> > > > > > > connector in that repo.
> > > > > > > - New connector will be in Flink repo and maintained like the
> > other
> > > > > > > connectors.
> > > > > > >
> > > > > > > This seems much simpler for users understand and they will not
> be
> > > > > blocked
> > > > > > > from using the old connector. If the concern is about the
> quality
> > > of
> > > > > the
> > > > > > > connector in streamnative/pulsar-flink repo, is it enough for
> us
> > > just
> > > > > to
> > > > > > > review the code in streamnative/pulsar-flink connector to make
> > sure
> > > > it
> > > > > > > looks good from Flink's perspective?
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]
> >
> > > > wrote:
> > > > > > >
> > > > > > > > My take would be the following:
> > > > > > > >
> > > > > > > >   - If we merge the connector now and replace it with a
> FLIP-27
> > > > > version
> > > > > > > > before the 1.10 release, then we need no deprecation process
> > > > > > > >   - If we don't manage to replace it with a FLIP-27 version
> > > before
> > > > > the
> > > > > > > 1.10
> > > > > > > > release, than it is good that we have the other version, so
> no
> > > > users
> > > > > > get
> > > > > > > > blocked.
> > > > > > > >
> > > > > > > > In the latter case we can see how we want to do it. Immediate
> > > > removal
> > > > > > of
> > > > > > > > the old version or deprecation label and keeping it for one
> > more
> > > > > > release.
> > > > > > > > Given that you should be able to use a Flink 1.10 connector
> > with
> > > > > Flink
> > > > > > > 1.11
> > > > > > > > as well (stable public APIs) there is also a workaround if
> you
> > > need
> > > > > an
> > > > > > > old
> > > > > > > > connector in a newer version. So immediate removal might even
> > be
> > > > > > > feasible.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Stephan,
> > > > > > > > >
> > > > > > > > > Thanks for the clarification. I completely agree with you
> and
> > > > > Thomas
> > > > > > on
> > > > > > > > the
> > > > > > > > > process of adding connectors to Flink repo. However, I am
> > > > wondering
> > > > > > > what
> > > > > > > > is
> > > > > > > > > the deprecation process? Given the main concern here was
> that
> > > we
> > > > > may
> > > > > > > have
> > > > > > > > > to maintain two Pulsar connector code bases until the old
> one
> > > is
> > > > > > > removed
> > > > > > > > > from the repo, it would be good to know how long we have to
> > do
> > > > > that.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Some quick thoughts on the connector contribution
> process.
> > I
> > > > > > > basically
> > > > > > > > > > reiterate here what Thomas mentioned in another thread
> > about
> > > > the
> > > > > > > > Kinesis
> > > > > > > > > > connector.
> > > > > > > > > >
> > > > > > > > > > For connectors, we should favor a low-overhead
> contribution
> > > > > > process,
> > > > > > > > and
> > > > > > > > > > accept user code and changes more readily than in the
> core
> > > > > system.
> > > > > > > > > > That is because connectors have both a big variety of
> > > scenarios
> > > > > > they
> > > > > > > > get
> > > > > > > > > > used in (only through use and many small contributions do
> > > they
> > > > > > become
> > > > > > > > > > really useful over time) and at the same time, and
> > committers
> > > > do
> > > > > > not
> > > > > > > > use
> > > > > > > > > > the connector themselves and usually cannot foresee too
> > well
> > > > what
> > > > > > is
> > > > > > > > > > needed.
> > > > > > > > > >
> > > > > > > > > > Further more, a missing connector (or connector feature)
> is
> > > > > often a
> > > > > > > > > bigger
> > > > > > > > > > show stopper for users than a missing API or system
> > feature.
> > > > > > > > > >
> > > > > > > > > > Along these lines of thougt, the conclusion would be to
> > take
> > > > the
> > > > > > > Pulsar
> > > > > > > > > > connector now, focus the review on
> legal/dependencies/rough
> > > > code
> > > > > > > style
> > > > > > > > > and
> > > > > > > > > > conventions, label it as "beta" (in the sense of "new
> code"
> > > > that
> > > > > is
> > > > > > > > "not
> > > > > > > > > > yet tested through longer use") and go ahead. And then
> > evolve
> > > > it
> > > > > > > > quickly
> > > > > > > > > > without putting formal blockers in the way, meaning also
> > > > adding a
> > > > > > new
> > > > > > > > > FLIP
> > > > > > > > > > 27 version when it is there.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yijie,
> > > > > > > > > > >
> > > > > > > > > > > Could you please follow the FLIP process to start a new
> > > FLIP
> > > > > > > > > [DISCUSSION]
> > > > > > > > > > > thread in the mailing list?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > > > > > > > > >
> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now.
> So
> > > > there
> > > > > > is a
> > > > > > > > > FLIP
> > > > > > > > > > > number collision. Can you change the FLIP number to 72?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for sharing the pulsar FLIP.
> > > > > > > > > > > > Would you mind enabling comments/suggestions on the
> > > google
> > > > > doc
> > > > > > > > link?
> > > > > > > > > > This
> > > > > > > > > > > > way the contributors from the community can comment
> on
> > > the
> > > > > doc.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Rong
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hello everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've drafted a FLIP that describes the current
> design
> > > of
> > > > > the
> > > > > > > > Pulsar
> > > > > > > > > > > > > connector:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please take a look and let me know what you think.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Yijie
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi All,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Sorry for joining the discussion late and thanks
> > > Yijie
> > > > &
> > > > > > > Sijie
> > > > > > > > > for
> > > > > > > > > > > > > driving
> > > > > > > > > > > > > > the discussion.
> > > > > > > > > > > > > > I also think the Pulsar connector would be a very
> > > > > valuable
> > > > > > > > > addition
> > > > > > > > > > > to
> > > > > > > > > > > > > > Flink. I can also help out a bit on the review
> side
> > > :-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the timeline, I also share concerns
> with
> > > > Becket
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > > > relationship between the new Pulsar connector and
> > > > > FLIP-27.
> > > > > > > > > > > > > > There's also another discussion just started by
> > > Stephan
> > > > > on
> > > > > > > > > dropping
> > > > > > > > > > > > Kafka
> > > > > > > > > > > > > > 9/10 support on next Flink release [1].  Although
> > the
> > > > > > > situation
> > > > > > > > > is
> > > > > > > > > > > > > somewhat
> > > > > > > > > > > > > > different, and Kafka 9/10 connector has been in
> > Flink
> > > > for
> > > > > > > > almost
> > > > > > > > > > 3-4
> > > > > > > > > > > > > years,
> > > > > > > > > > > > > > based on the discussion I am not sure if a major
> > > > version
> > > > > > > > release
> > > > > > > > > > is a
> > > > > > > > > > > > > > requirement for removing old connector supports.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree
> > the
> > > > old
> > > > > > > > > connector
> > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is
> > there.
> > > > As
> > > > > > > > Stephan
> > > > > > > > > > > > stated,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > is easier to contribute the source sooner and
> > adjust
> > > it
> > > > > > > later.
> > > > > > > > > > > > > > We should also ensure we clearly communicate the
> > > > message:
> > > > > > for
> > > > > > > > > > > example,
> > > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27
> > > > connector
> > > > > > page
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > website, documentations, etc. Any other thoughts?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Rong
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Technically speaking, removing the old
> connector
> > > code
> > > > > is
> > > > > > a
> > > > > > > > > > > backwards
> > > > > > > > > > > > > > > incompatible change which requires a major
> > version
> > > > > bump,
> > > > > > > i.e.
> > > > > > > > > > Flink
> > > > > > > > > > > > > 2.x.
> > > > > > > > > > > > > > > Given that we don't have a clear plan on when
> to
> > > have
> > > > > the
> > > > > > > > next
> > > > > > > > > > > major
> > > > > > > > > > > > > > > version release, it seems unclear how long the
> > old
> > > > > > > connector
> > > > > > > > > code
> > > > > > > > > > > > will
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > there if we check it in right now. Or will we
> > > remove
> > > > > the
> > > > > > > old
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > without a major version bump? In any case, it
> > > sounds
> > > > > not
> > > > > > > > quite
> > > > > > > > > > user
> > > > > > > > > > > > > > > friendly to the those who might use the old
> > Pulsar
> > > > > > > > connector. I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > if it is worth these potential problems in
> order
> > to
> > > > > have
> > > > > > > the
> > > > > > > > > > Pulsar
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > connector checked in one or two months earlier.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
> > > > > > > > [hidden email]
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Agreed, if we check in the old code, we
> should
> > > make
> > > > > it
> > > > > > > > clear
> > > > > > > > > > that
> > > > > > > > > > > > it
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based
> version
> > > of
> > > > > the
> > > > > > > > > > connector
> > > > > > > > > > > is
> > > > > > > > > > > > > > > there.
> > > > > > > > > > > > > > > > We should not commit to maintaining the old
> > > > version,
> > > > > > that
> > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > > > indeed
> > > > > > > > > > > > > > > > too much overhead.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi Stephan,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for the volunteering to help.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Yes, the overhead would just be review
> > > capacity.
> > > > In
> > > > > > > > fact, I
> > > > > > > > > > am
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > worrying
> > > > > > > > > > > > > > > > > too much about the review capacity. That is
> > > just
> > > > a
> > > > > > one
> > > > > > > > time
> > > > > > > > > > > cost.
> > > > > > > > > > > > > My
> > > > > > > > > > > > > > > > > concern is mainly about the long term
> burden.
> > > > > Assume
> > > > > > we
> > > > > > > > > have
> > > > > > > > > > > new
> > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly added
> > Pulsar
> > > > > > > > connectors
> > > > > > > > > in
> > > > > > > > > > > old
> > > > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to
> > new
> > > > > > source
> > > > > > > > > > > interface,
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > old
> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated almost
> > > > > > immediately
> > > > > > > > > after
> > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > in,
> > > > > > > > > > > > > > > > > but we may still have to maintain two code
> > > bases.
> > > > > For
> > > > > > > the
> > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > connectors, we have to do that anyways. But
> > it
> > > > > would
> > > > > > be
> > > > > > > > > good
> > > > > > > > > > to
> > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > introducing a new connector with the same
> > > > problem.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan
> Ewen
> > <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Nice to see this lively discussion about
> > the
> > > > > Pulsar
> > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > Some thoughts on the open questions:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a
> > > > community
> > > > > > > > package
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Looks like the discussion is more going
> > > towards
> > > > > > > > > > > contribution. I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > is good, especially if we think that we
> > want
> > > to
> > > > > > > build a
> > > > > > > > > > > > similarly
> > > > > > > > > > > > > > > deep
> > > > > > > > > > > > > > > > > > integration with Pulsar as we have for
> > > example
> > > > > with
> > > > > > > > > Kafka.
> > > > > > > > > > > The
> > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > already looks like a more thorough
> > connector
> > > > than
> > > > > > > many
> > > > > > > > > > others
> > > > > > > > > > > > we
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the repository.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > With either a repo split, or the new
> build
> > > > > system,
> > > > > > I
> > > > > > > > hope
> > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > > overhead is not a problem.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > ## Committer Support
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Becket offered some help already, I can
> > also
> > > > > help a
> > > > > > > > bit.
> > > > > > > > > I
> > > > > > > > > > > hope
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > between us, we can cover this.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually
> making
> > > some
> > > > > > > PoC-ing
> > > > > > > > > > > > > progress, but
> > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > take 2 more months, I would estimate,
> > before
> > > it
> > > > > is
> > > > > > > > fully
> > > > > > > > > > > > > available.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > If we want to be on the safe side with
> the
> > > > > > > > contribution,
> > > > > > > > > we
> > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > contribute the source sooner and adjust
> it
> > > > later.
> > > > > > > That
> > > > > > > > > > would
> > > > > > > > > > > > also
> > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > > > > in case things get crazy towards the 1.10
> > > > feature
> > > > > > > > freeze
> > > > > > > > > > and
> > > > > > > > > > > it
> > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > hard to find time to review the new
> > changes.
> > > > > > > > > > > > > > > > > > What would be the overhead of
> contributing
> > > now?
> > > > > > Given
> > > > > > > > > that
> > > > > > > > > > > the
> > > > > > > > > > > > > code
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > already there, it looks like it would be
> > only
> > > > > > review
> > > > > > > > > > > capacity,
> > > > > > > > > > > > > right?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie
> > Shen <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hi everyone!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your attention and the
> > promotion
> > > > of
> > > > > > this
> > > > > > > > > work.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as
> > possible
> > > > for
> > > > > > more
> > > > > > > > > > > specific
> > > > > > > > > > > > > > > > > discussions.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not
> > > > reached
> > > > > a
> > > > > > > > > > consensus.
> > > > > > > > > > > > > > > > Therefore,
> > > > > > > > > > > > > > > > > > > I will explain all the functionalities
> of
> > > the
> > > > > > > > existing
> > > > > > > > > > > > > connector in
> > > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and
> > > > Catalog)
> > > > > to
> > > > > > > > > > continue
> > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > discussions in FLIP.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks for your kind help.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket
> > Qin
> > > <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Sijie,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have
> > > Pulsar
> > > > > > > > connector
> > > > > > > > > > in
> > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > do the following:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar
> connector
> > > to
> > > > > > Flink
> > > > > > > > main
> > > > > > > > > > > repo
> > > > > > > > > > > > > as it
> > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > public interface to Flink main repo.
> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink
> > right
> > > > away
> > > > > > as
> > > > > > > > > there
> > > > > > > > > > is
> > > > > > > > > > > > no
> > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > sink interface so far.
> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27.
> Flink
> > > 1.10
> > > > > is
> > > > > > > > going
> > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > > freeze
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > late Nov and let's say we give a
> month
> > to
> > > > the
> > > > > > > > > > development
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have
> > FLIP-27
> > > > by
> > > > > > late
> > > > > > > > > Oct.
> > > > > > > > > > > > There
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > 7
> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it is
> doable.
> > > If
> > > > > > > FLIP-27
> > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > > > > ready
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > late
> > > > > > > > > > > > > > > > > > > > Oct, we can review and check in
> Pulsar
> > > > > > connector
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > interface. This means we will have
> > Pulsar
> > > > > > > connector
> > > > > > > > > in
> > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > either
> > > > > > > > > > > > > > > > > > > > with or without FLIP-27.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar
> > sink
> > > > and
> > > > > > > source
> > > > > > > > > > > checked
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > separately,
> > > > > > > > > > > > > > > > > > > > it might make sense to have two
> FLIPs,
> > > one
> > > > > for
> > > > > > > > Pulsar
> > > > > > > > > > > sink
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > another
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can start the
> > work
> > > on
> > > > > > > Pulsar
> > > > > > > > > sink
> > > > > > > > > > > > right
> > > > > > > > > > > > > > > away.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie
> > Guo
> > > <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > What's the take from Flink
> community?
> > > > Shall
> > > > > > we
> > > > > > > > wait
> > > > > > > > > > for
> > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > shall we
> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And what the
> > > next
> > > > > > steps
> > > > > > > > are?
> > > > > > > > > > :-)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM
> Bowen
> > > Li <
> > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector
> > in
> > > > > Flink
> > > > > > > can
> > > > > > > > > be a
> > > > > > > > > > > > good
> > > > > > > > > > > > > > > mutual
> > > > > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > both communities.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Another perspective is that
> Pulsar
> > > > > > connector
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > > > 1st
> > > > > > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > that integrates with Flink's
> > metadata
> > > > > > > > management
> > > > > > > > > > > system
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > Catalog
> > > > > > > > > > > > > > > > > > > APIs.
> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the
> > > > integration
> > > > > > > turns
> > > > > > > > > out
> > > > > > > > > > > and
> > > > > > > > > > > > > > > whether
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > need to
> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack,
> which
> > > are
> > > > > > > > currently
> > > > > > > > > in
> > > > > > > > > > > > > Beta, to
> > > > > > > > > > > > > > > > > cater
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm
> in
> > > > favor
> > > > > of
> > > > > > > > > merging
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > > > > > Flink 1.10.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller
> sized
> > > > PRs,
> > > > > > e.g.
> > > > > > > > > maybe
> > > > > > > > > > > one
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > basic
> > > > > > > > > > > > > > > > > > > > > > source/sink functionalities and
> > > another
> > > > > for
> > > > > > > > > schema
> > > > > > > > > > > and
> > > > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > > > > > > > integration,
> > > > > > > > > > > > > > > > > > > > > > just to make them easier to
> review.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait
> for
> > > > > > FLIP-27.
> > > > > > > > But
> > > > > > > > > I
> > > > > > > > > > > > don't
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > FLIP-27
> > > > > > > > > > > > > > > > > > > > > > should be a blocker in cases
> where
> > it
> > > > > > cannot
> > > > > > > > make
> > > > > > > > > > its
> > > > > > > > > > > > way
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > 1.10
> > > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount
> of
> > > time
> > > > > for
> > > > > > > > > > > committers
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > review
> > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt
> to
> > > new
> > > > > > > > > interfaces.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Bowen
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM
> > Becket
> > > > > Qin <
> > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Hi Till,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends
> on
> > > when
> > > > > the
> > > > > > > new
> > > > > > > > > > > source
> > > > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it
> > > would
> > > > > be
> > > > > > > > there
> > > > > > > > > in
> > > > > > > > > > > > > about a
> > > > > > > > > > > > > > > > month
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > so.
> > > > > > > > > > > > > > > > > > > > > > But
> > > > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It
> > would
> > > > > also
> > > > > > be
> > > > > > > > > good
> > > > > > > > > > to
> > > > > > > > > > > > > hear
> > > > > > > > > > > > > > > what
> > > > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as they are
> > also
> > > > > > involved
> > > > > > > > in
> > > > > > > > > > > > FLIP-27.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > In general I think we should
> have
> > > > > Pulsar
> > > > > > > > > > connector
> > > > > > > > > > > in
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > 1.10,
> > > > > > > > > > > > > > > > > > > > > > > preferably with the new source
> > > > > interface.
> > > > > > > We
> > > > > > > > > can
> > > > > > > > > > > also
> > > > > > > > > > > > > check
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > right
> > > > > > > > > > > > > > > > > > > > > > now
> > > > > > > > > > > > > > > > > > > > > > > with old source interface, but
> I
> > > > > suspect
> > > > > > > few
> > > > > > > > > > users
> > > > > > > > > > > > > will use
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > before
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > next official release.
> Therefore,
> > > it
> > > > > > seems
> > > > > > > > > > > reasonable
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > wait a
> > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > to see whether we can jump to
> the
> > > new
> > > > > > > source
> > > > > > > > > > > > > interface. As
> > > > > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it,
> waiting a
> > > > > little
> > > > > > > bit
> > > > > > > > > > > doesn't
> > > > > > > > > > > > > seem
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > hurt
> > > > > > > > > > > > > > > > > > > much.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM
> > Till
> > > > > > > Rohrmann
> > > > > > > > <
> > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the
> problem
> > > > would
> > > > > be
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > > > > committed
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > connector before the new
> source
> > > > > > interface
> > > > > > > > is
> > > > > > > > > > > ready.
> > > > > > > > > > > > > If I
> > > > > > > > > > > > > > > > > > > understood
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to
> > > support
> > > > > the
> > > > > > > old
> > > > > > > > > > source
> > > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > > > > anyway
> > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > existing connectors. By
> > checking
> > > it
> > > > > in
> > > > > > > > early
> > > > > > > > > I
> > > > > > > > > > > > could
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > benefit
> > > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > > > our users could start using
> the
> > > > > > connector
> > > > > > > > > > > earlier.
> > > > > > > > > > > > > > > > Moreover,
> > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar
> > > integration
> > > > > is
> > > > > > > > being
> > > > > > > > > > > > delayed
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > > that the
> > > > > > > > > > > > > > > > > > > > > > > > source interface should be
> > > delayed.
> > > > > The
> > > > > > > > only
> > > > > > > > > > > > > downside I
> > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > extra
> > > > > > > > > > > > > > > > > > > > > > > > review effort and potential
> > fixes
> > > > > which
> > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > > > > > > irrelevant
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > source interface
> > implementation.
> > > I
> > > > > > guess
> > > > > > > it
> > > > > > > > > > > mainly
> > > > > > > > > > > > > > > depends
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > > certain
> > > > > > > > > > > > > > > > > > > > > > > > we are when the new source
> > > > interface
> > > > > > will
> > > > > > > > be
> > > > > > > > > > > ready.
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56
> AM
> > > > Becket
> > > > > > > Qin <
> > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your
> > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some
> update
> > > on
> > > > > > > FLIP-27.
> > > > > > > > > > > > Although
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread has been
> > > quiet
> > > > > for
> > > > > > > some
> > > > > > > > > > > time, a
> > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > committer /
> > > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink
> > community
> > > > > were
> > > > > > > > > actually
> > > > > > > > > > > > > > > prototyping
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > entire
> > > > > > > > > > > > > > > > > > > > > > > > thing.
> > > > > > > > > > > > > > > > > > > > > > > > > We have made some good
> > progress
> > > > > there
> > > > > > > but
> > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > > > update
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > FLIP
> > > > > > > > > > > > > > > > > > > > > > wiki
> > > > > > > > > > > > > > > > > > > > > > > > > after the entire thing is
> > > > verified
> > > > > to
> > > > > > > > work
> > > > > > > > > in
> > > > > > > > > > > > case
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > last
> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the
> > > > > > implementation.
> > > > > > > I
> > > > > > > > > > don't
> > > > > > > > > > > > > have an
> > > > > > > > > > > > > > > > > exact
> > > > > > > > > > > > > > > > > > > ETA
> > > > > > > > > > > > > > > > > > > > > yet,
> > > > > > > > > > > > > > > > > > > > > > > > but I
> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be
> > within
> > > a
> > > > > > month
> > > > > > > or
> > > > > > > > > so.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the
> > > current
> > > > > > Flink
> > > > > > > > > Pulsar
> > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > see if
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be
> > > good
> > > > to
> > > > > > > avoid
> > > > > > > > > the
> > > > > > > > > > > > case
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > checked
> > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some
> > > review
> > > > > > > efforts
> > > > > > > > > and
> > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is ready.
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39
> > AM
> > > > > Yijie
> > > > > > > > Shen <
> > > > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the
> feedback
> > > and
> > > > > > > > > > suggestions!
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal
> of
> > > the
> > > > > > > > connector
> > > > > > > > > > has
> > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > > > users with the latest
> > > features
> > > > of
> > > > > > > both
> > > > > > > > > > > systems
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > possible.
> > > > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the
> > > > > connector
> > > > > > > to
> > > > > > > > > > Flink
> > > > > > > > > > > > and
> > > > > > > > > > > > > hope
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback
> > from
> > > > > Flink
> > > > > > > > > experts
> > > > > > > > > > > to
> > > > > > > > > > > > > ensure
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > high
> > > > > > > > > > > > > > > > > > > > > > > quality
> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed
> its
> > > > > > existence
> > > > > > > > at
> > > > > > > > > > the
> > > > > > > > > > > > > > > beginning
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > reworking
> > > > > > > > > > > > > > > > > > > > > > > > > > the connector
> > implementation
> > > > > based
> > > > > > on
> > > > > > > > > Flink
> > > > > > > > > > > > 1.9;
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > wanted
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector that
> > > supports
> > > > > > both
> > > > > > > > > batch
> > > > > > > > > > > and
> > > > > > > > > > > > > stream
> > > > > > > > > > > > > > > > > > > computing
> > > > > > > > > > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > > > > > > > > > > > on it.
> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has been
> > inactive
> > > > for
> > > > > > > some
> > > > > > > > > > time,
> > > > > > > > > > > so
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > decided
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with most of
> > the
> > > > new
> > > > > > > > > features,
> > > > > > > > > > > such
> > > > > > > > > > > > > as
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > type
> > > > > > > > > > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API
> > > first.
> > > > We
> > > > > > > will
> > > > > > > > > pay
> > > > > > > > > > > > > attention
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > progress
> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually
> and
> > > > > > > incorporate
> > > > > > > > it
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > soon
> > > > > > > > > > > > > > > > > > > > > > > > > > as possible.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status
> > of
> > > > the
> > > > > > > > > connector,
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
> > > > > > repository
> > > > > > > > and
> > > > > > > > > > > aimed
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > provide
> > > > > > > > > > > > > > > > > > > > > > throughout
> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are
> > > also
> > > > > > happy
> > > > > > > to
> > > > > > > > > > hear
> > > > > > > > > > > > > > > > suggestions
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from the
> Flink
> > > > > > community
> > > > > > > to
> > > > > > > > > > > improve
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > stability and
> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the
> > connector
> > > > > > > > > continuously.
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> 5:59
> > > AM
> > > > > > Sijie
> > > > > > > > Guo
> > > > > > > > > <
> > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the
> > > > > comments
> > > > > > > and
> > > > > > > > > > > > feedback.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the
> > > main
> > > > > > > question
> > > > > > > > > > here
> > > > > > > > > > > is
> > > > > > > > > > > > > > > about -
> > > > > > > > > > > > > > > > > > "how
> > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > community maintain the
> > > > > > connector?".
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts
> > from
> > > > > > myself.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and
> where
> > to
> > > > > host
> > > > > > > this
> > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > kind
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > less
> > > > > > > > > > > > > > > > > > > > > > > > > > important
> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there
> can
> > > be
> > > > > many
> > > > > > > > ways
> > > > > > > > > to
> > > > > > > > > > > > > achieve
> > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the
> > > contribution,
> > > > > what
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > looking
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > here
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > > > > these
> > > > > > > > > > > > > > > > > > > > > > > > > > two
> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can build
> the
> > > > > > > > collaboration
> > > > > > > > > > > > > > > relationship
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > developing
> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration between
> > > > Pulsar
> > > > > > and
> > > > > > > > > Flink.
> > > > > > > > > > > > Even
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > try
> > > > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > catch
> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in
> > Flink
> > > > > > > > community.
> > > > > > > > > We
> > > > > > > > > > > are
> > > > > > > > > > > > > still
> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we
> > > have
> > > > > less
> > > > > > > > > > > experiences
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > than
> > > > > > > > > > > > > > > > > > > > > folks
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In order to
> > make
> > > > > sure
> > > > > > we
> > > > > > > > > > > maintain
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > deliver
> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality
> > pulsar-flink
> > > > > > > > integration
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > who
> > > > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need
> > some
> > > > help
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > > > experts
> > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > community.
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been
> following
> > > > > FLIP-27
> > > > > > > > for a
> > > > > > > > > > > > while.
> > > > > > > > > > > > > > > > > Originally
> > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > were
> > > > > > > > > > > > > > > > > > > > > > > > > > thinking
> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the
> > > > connectors
> > > > > > back
> > > > > > > > > after
> > > > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in
> > > > FLIP-27.
> > > > > > But
> > > > > > > we
> > > > > > > > > > > decided
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > initiate
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > conversation
> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as possible.
> > > Because
> > > > > we
> > > > > > > > > believe
> > > > > > > > > > > > there
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > benefits
> > > > > > > > > > > > > > > > > > > > > > > > doing
> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than
> later.
> > > As
> > > > > part
> > > > > > > of
> > > > > > > > > > > > > contribution,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > community understand
> more
> > > > about
> > > > > > > > Pulsar
> > > > > > > > > > and
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > potential
> > > > > > > > > > > > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > > > > > > > > > > points.
> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help
> > Flink
> > > > > > > community
> > > > > > > > > > > verify
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g.
> > > > catalog
> > > > > > > API).
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
> > 5:24
> > > > AM
> > > > > > > Becket
> > > > > > > > > > Qin <
> > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> interest
> > > in
> > > > > > > > > contributing
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > connector.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think
> > > having
> > > > > > Pulsar
> > > > > > > > > > > connector
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > strong
> > > > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to
> > > Flink.
> > > > > So
> > > > > > I
> > > > > > > am
> > > > > > > > > > happy
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > shepherd
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > effort.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would
> also
> > > > like
> > > > > to
> > > > > > > > > provide
> > > > > > > > > > > > some
> > > > > > > > > > > > > > > > context
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > recent
> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors
> > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink
> > > > > maintains
> > > > > > > its
> > > > > > > > > > > > > connector has
> > > > > > > > > > > > > > > > hit
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > > > > > > > > > > > bar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more
> > > > connectors
> > > > > > > > coming
> > > > > > > > > > into
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > repo,
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > > > facing a
> > > > > > > > > > > > > > > > > > > > > > > > > > few
> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long
> > > build
> > > > > and
> > > > > > > > > testing
> > > > > > > > > > > > > time. To
> > > > > > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > > > > problem,
> > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do
> > the
> > > > > > > following:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the
> > > connectors
> > > > > > into
> > > > > > > a
> > > > > > > > > > > separate
> > > > > > > > > > > > > > > > > repository.
> > > > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily
> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to
> > potential
> > > > > > solution
> > > > > > > > to
> > > > > > > > > > > > shorten
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > build
> > > > > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the
> > > connectors
> > > > > to
> > > > > > > stay
> > > > > > > > > as
> > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > tries
> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good
> support
> > > for
> > > > > > > > > > functionality
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > > > > > > tests.
> > > > > > > > > > > > > > > > > > > > > > > > > > Robert
> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create
> a
> > > > Flink
> > > > > > > > > Ecosystem
> > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > website
> > > > > > > > > > > > > > > > > > > and it
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some final
> > > approval
> > > > > > > > process.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above
> > efforts,
> > > it
> > > > > > would
> > > > > > > > be
> > > > > > > > > > > great
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > if we
> > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as
> an
> > > > > > ecosystem
> > > > > > > > > > project
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > great
> > > > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the
> > > Flink
> > > > > > Pulsar
> > > > > > > > > > > connector
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > tested
> > > > > > > > > > > > > > > > > > > > > currently
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > see
> > > > > > > > > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn
> something
> > to
> > > > > > > maintain
> > > > > > > > it
> > > > > > > > > > as
> > > > > > > > > > > an
> > > > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test
> > > coverage.
> > > > If
> > > > > > the
> > > > > > > > > > quality
> > > > > > > > > > > > as
> > > > > > > > > > > > > an
> > > > > > > > > > > > > > > > > > ecosystem
> > > > > > > > > > > > > > > > > > > > > > project
> > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > hard
> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may
> as
> > > > well
> > > > > > > adopt
> > > > > > > > it
> > > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > main
> > > > > > > > > > > > > > > > > > repo.
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing
> > > effort
> > > > > is
> > > > > > > > > FLIP-27
> > > > > > > > > > > > where
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source
> connector
> > > > > > > architecture
> > > > > > > > > and
> > > > > > > > > > > > > > > interface.
> > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > > > > > > > > likely
> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10.
> Therefore
> > > > > timing
> > > > > > > > wise,
> > > > > > > > > if
> > > > > > > > > > > we
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main
> > repo, I
> > > > am
> > > > > > > > > wondering
> > > > > > > > > > if
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > hold
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > little
> > > > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar
> > connector
> > > > > adapt
> > > > > > to
> > > > > > > > the
> > > > > > > > > > new
> > > > > > > > > > > > > > > interface
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > avoid
> > > > > > > > > > > > > > > > > > > > > > > > shortly
> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019
> at
> > > 4:32
> > > > > PM
> > > > > > > > > Chesnay
> > > > > > > > > > > > > Schepler <
> > > > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried
> > that
> > > we
> > > > > may
> > > > > > > end
> > > > > > > > > up
> > > > > > > > > > > > > repeating
> > > > > > > > > > > > > > > > > > history.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already
> 2
> > > > > attempts
> > > > > > > at
> > > > > > > > > > > > > contributing a
> > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed
> > because
> > > > no
> > > > > > > > > committer
> > > > > > > > > > > was
> > > > > > > > > > > > > > > getting
> > > > > > > > > > > > > > > > > > > involved,
> > > > > > > > > > > > > > > > > > > > > > > > despite
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor
> opening a
> > > > > > dedicated
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > > > about the
> > > > > > > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and
> > getting
> > > > > > several
> > > > > > > > +1's
> > > > > > > > > > > from
> > > > > > > > > > > > > > > > > committers.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really
> make
> > > > sure
> > > > > > that
> > > > > > > > if
> > > > > > > > > we
> > > > > > > > > > > > > > > > > welcome/approve
> > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it
> will
> > > > > actually
> > > > > > > get
> > > > > > > > > the
> > > > > > > > > > > > > attention
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > deserves.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm
> inclined
> > > to
> > > > > > > > recommend
> > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > outside
> > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could
> link
> > to
> > > > it
> > > > > > from
> > > > > > > > the
> > > > > > > > > > > > > > > documentation
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming
> > page
> > > > for
> > > > > > > > sharing
> > > > > > > > > > > > > artifacts
> > > > > > > > > > > > > > > > among
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > > > > (what's
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that
> > > > anyway?),
> > > > > > > this
> > > > > > > > > may
> > > > > > > > > > > be a
> > > > > > > > > > > > > > > better
> > > > > > > > > > > > > > > > > > > option.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019
> 10:16,
> > > Till
> > > > > > > > Rohrmann
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for
> > > > starting
> > > > > > > this
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > Yijie.
> > > > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would
> be
> > a
> > > > very
> > > > > > > > > valuable
> > > > > > > > > > > > > addition
> > > > > > > > > > > > > > > > since
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > becomes
> > > > > > > > > > > > > > > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and
> it
> > > > would
> > > > > > > > further
> > > > > > > > > > > > expand
> > > > > > > > > > > > > > > > Flink's
> > > > > > > > > > > > > > > > > > > > > > > > > interoperability.
> > > > > > > > > > > > > > > > > > > > > > > > > > Also
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project
> > > > > perspective
> > > > > > it
> > > > > > > > > makes
> > > > > > > > > > > > > sense for
> > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > place
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream
> > > project.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> > > > concern/question
> > > > > is
> > > > > > > how
> > > > > > > > > can
> > > > > > > > > > > the
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > > maintain
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We
> have
> > > seen
> > > > > in
> > > > > > > the
> > > > > > > > > past
> > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > most
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively
> developed
> > > > > > components
> > > > > > > > > > because
> > > > > > > > > > > > > they
> > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > sync
> > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external system
> and
> > > > with
> > > > > > > Flink.
> > > > > > > > > > Given
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > community
> > > > > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
> > > > maintaining,
> > > > > > > > > improving
> > > > > > > > > > > and
> > > > > > > > > > > > > > > > evolving
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > connector,
> > > > > > > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that
> we
> > > can
> > > > > > > achieve
> > > > > > > > > > this.
> > > > > > > > > > > > > Hence,
> > > > > > > > > > > > > > > +1
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> 2019
> > > at
> > > > > 2:03
> > > > > > > AM
> > > > > > > > > > Sijie
> > > > > > > > > > > > Guo
> > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the
> > main
> > > > > > driver
> > > > > > > > > behind
> > > > > > > > > > > > > > > FLINK-9641
> > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168,
> > > > > > > > > > > > > > > > > > > > > > > > > let
> > > > > > > > > > > > > > > > > > > > > > > > > > me
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context
> > on
> > > > > this.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and
> > > > > FLINK-9168
> > > > > > > was
> > > > > > > > > > > created
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > bringing
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink.
> > The
> > > > > > > > integration
> > > > > > > > > > was
> > > > > > > > > > > > done
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > 1.6.0.
> > > > > > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > > > > > sent out
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a
> > > year
> > > > > ago
> > > > > > > and
> > > > > > > > we
> > > > > > > > > > > ended
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > maintaining
> > > > > > > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > > > > > > connectors
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for
> Pulsar
> > > > users
> > > > > to
> > > > > > > use
> > > > > > > > > > Flink
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > ).
> > > > > > > > > > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration
> is
> > > > > pretty
> > > > > > > > simple
> > > > > > > > > > and
> > > > > > > > > > > > > there
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past
> year,
> > we
> > > > > have
> > > > > > > > made a
> > > > > > > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > brought
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as
> > the
> > > > > > > > first-class
> > > > > > > > > > > > > citizen in
> > > > > > > > > > > > > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > We
> > > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > integrated
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing
> > > > engines
> > > > > > for
> > > > > > > > > > > processing
> > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > streams
> > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to
> > rethink
> > > > how
> > > > > > to
> > > > > > > > > > > integrate
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > best
> > > > > > > > > > > > > > > > > > > > > > > > > way.
> > > > > > > > > > > > > > > > > > > > > > > > > > Then
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
> > > > > > pulsar-flink
> > > > > > > > > > > > connectors
> > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > ground
> > > > > > > > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > schema
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API
> > and
> > > > > > catalog
> > > > > > > > API
> > > > > > > > > as
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > > > > citizen
> > > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration.
> With
> > > that
> > > > > > being
> > > > > > > > > said,
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register
> > > > pulsar
> > > > > > as a
> > > > > > > > > flink
> > > > > > > > > > > > > catalog
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > query /
> > > > > > > > > > > > > > > > > > > > > > > process
> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > event
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using
> > Flink
> > > > SQL.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an
> example
> > > > about
> > > > > > how
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > catalog:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also
> > > > written a
> > > > > > > blog
> > > > > > > > > post
> > > > > > > > > > > > > > > explaining
> > > > > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > > > > re-implement
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with
> > Flink
> > > > 1.9
> > > > > > and
> > > > > > > > > what
> > > > > > > > > > > are
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > made
> > > > > > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe
> Pulsar
> > is
> > > > not
> > > > > > > just
> > > > > > > > a
> > > > > > > > > > > simple
> > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > sink
> > > > > > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > > > > > > source
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be
> a
> > > > fully
> > > > > > > > > integrated
> > > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink,
> > source,
> > > > > > > > > > schema/catalog
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > state).
> > > > > > > > > > > > > > > > > > The
> > > > > > > > > > > > > > > > > > > > > > > > combination
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can
> > > create
> > > > a
> > > > > > > great
> > > > > > > > > > > > streaming
> > > > > > > > > > > > > > > > > warehouse
> > > > > > > > > > > > > > > > > > > > > > > architecture
> > > > > > > > > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first,
> > > > unified
> > > > > > > data
> > > > > > > > > > > > > processing.
> > > > > > > > > > > > > > > > Since
> > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > > talking
> > > > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute
> Pulsar
> > > > > > > integration
> > > > > > > > to
> > > > > > > > > > > Flink
> > > > > > > > > > > > > here,
> > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain,
> improve
> > > and
> > > > > > evolve
> > > > > > > > the
> > > > > > > > > > > > > integration
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both
> Flink
> > > and
> > > > > > > Pulsar.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give
> > you a
> > > > bit
> > > > > > > more
> > > > > > > > > > > > background
> > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let
> > me
> > > > know
> > > > > > > what
> > > > > > > > > are
> > > > > > > > > > > your
> > > > > > > > > > > > > > > > thoughts.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3,
> > 2019
> > > at
> > > > > > 11:54
> > > > > > > > AM
> > > > > > > > > > Yun
> > > > > > > > > > > > > Tang <
> > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that
> > > Pulsar
> > > > > > > becomes
> > > > > > > > > > more
> > > > > > > > > > > > and
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > > popular
> > > > > > > > > > > > > > > > > > > > > > > recently
> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more
> > people
> > > > > > willing
> > > > > > > to
> > > > > > > > > > > > > contribute to
> > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > ecosystem.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any
> > further
> > > > > > > > discussion,
> > > > > > > > > > > would
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > please
> > > > > > > > > > > > > > > > > > > give
> > > > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > > > > > explanation
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the
> relationship
> > > > > between
> > > > > > > this
> > > > > > > > > > > thread
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > > > > > > > JIRAs
> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and
> > sink
> > > > [2]
> > > > > > > > > > connector?
> > > > > > > > > > > > > Will the
> > > > > > > > > > > > > > > > > > > > > contribution
> > > > > > > > > > > > > > > > > > > > > > > > > contains
> > > > > > > > > > > > > > > > > > > > > > > > > > > > part
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or
> > > totally
> > > > > > > > different
> > > > > > > > > > > > > > > > implementation?
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > > > > > >
> > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > ________________________________
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie
> Shen
> > <
> > > > > > > > > > > > > > > [hidden email]>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday,
> > > > > September
> > > > > > 3,
> > > > > > > > > 2019
> > > > > > > > > > > > 13:57
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> > > > > [hidden email]
> > > > > > <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject:
> > [DISCUSS]
> > > > > > > Contribute
> > > > > > > > > > > Pulsar
> > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > back
> > > > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink
> > > Community!
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to
> > > open
> > > > > the
> > > > > > > > > > discussion
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > contributing
> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0]
> > back
> > > to
> > > > > > > Flink.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
> > > > introduction
> > > > > > to
> > > > > > > > > Apache
> > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache
> Pulsar[1]
> > > is a
> > > > > > > > > > multi-tenant,
> > > > > > > > > > > > > > > > > > > high-performance
> > > > > > > > > > > > > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub
> messaging
> > > > > system.
> > > > > > > > > Pulsar
> > > > > > > > > > > > > includes
> > > > > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > > > > > > > such
> > > > > > > > > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support
> > for
> > > > > > multiple
> > > > > > > > > > > clusters
> > > > > > > > > > > > > in a
> > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > instance,
> > > > > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> > > > > geo-replication
> > > > > > of
> > > > > > > > > > > messages
> > > > > > > > > > > > > across
> > > > > > > > > > > > > > > > > > > clusters,
> > > > > > > > > > > > > > > > > > > > > > very
> > > > > > > > > > > > > > > > > > > > > > > > low
> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end
> > > > latency,
> > > > > > > > > seamless
> > > > > > > > > > > > > > > scalability
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > over a
> > > > > > > > > > > > > > > > > > > > > > > > million
> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed
> > > > message
> > > > > > > > delivery
> > > > > > > > > > > with
> > > > > > > > > > > > > > > > persistent
> > > > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > > > > > storage
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by
> > Apache
> > > > > > > > BookKeeper.
> > > > > > > > > > > > > Nowadays,
> > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > has
> > > > > > > > > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > > > > > > > > > > > adopted
> > > > > > > > > > > > > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
> > > > > > companies[2].
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status
> of
> > > > Pulsar
> > > > > > > Flink
> > > > > > > > > > > > connector
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar
> Flink
> > > > > > connector
> > > > > > > we
> > > > > > > > > are
> > > > > > > > > > > > > planning
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > contribute
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > > > > built
> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and
> > > > Pulsar
> > > > > > > 2.4.0.
> > > > > > > > > The
> > > > > > > > > > > > main
> > > > > > > > > > > > > > > > features
> > > > > > > > > > > > > > > > > > > are:
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a
> > > > streaming
> > > > > > > > source
> > > > > > > > > > with
> > > > > > > > > > > > > > > > > exactly-once
> > > > > > > > > > > > > > > > > > > > > > guarantee.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink
> streaming
> > > > > results
> > > > > > to
> > > > > > > > > > Pulsar
> > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > at-least-once
> > > > > > > > > > > > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > > > > > > > > > > (We
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update
> this
> > > to
> > > > > > > > > exactly-once
> > > > > > > > > > > as
> > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > > > > > > > all
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction
> > > features
> > > > > > ready
> > > > > > > in
> > > > > > > > > its
> > > > > > > > > > > > 2.5.0
> > > > > > > > > > > > > > > > > version)
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon
> > Flink
> > > > new
> > > > > > > Table
> > > > > > > > > API
> > > > > > > > > > > Type
> > > > > > > > > > > > > > > system
> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
> > > > > > (de)serialize
> > > > > > > > > > > messages
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > > > > > > > > > > > > > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate
> with
> > > > Flink
> > > > > > new
> > > > > > > > > > Catalog
> > > > > > > > > > > > API
> > > > > > > > > > > > > > > > > > > (FLIP-30[4]),
> > > > > > > > > > > > > > > > > > > > > > which
> > > > > > > > > > > > > > > > > > > > > > > > > > enables
> > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar
> > > topics
> > > > as
> > > > > > > > tables
> > > > > > > > > in
> > > > > > > > > > > > > Table API
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > well as
> > > > > > > > > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > > > > > > > > > client.
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > > > > > > > > > > > > > > >
> https://github.com/streamnative/pulsar-flink
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > > > > > > https://pulsar.apache.org/
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[郭士榕]

unscribe






At 2019-09-24 08:53:25, "Becket Qin" <[hidden email]> wrote:

>Thanks, Stephan.
>
>Sounds good to me. We can still try our best to get new Pulsar connector in
>Flink 1.10. In case we do not have time to do that, we will prominently
>link the Pulsar connector from the Flink connector docs.
>
>Thanks,
>
>Jiangjie (Becket) Qin
>
>On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <[hidden email]> wrote:
>
>> Okay, I see your point, Becket.
>>
>> Then let us prominently link the Pulsar connector from the Flink connector
>> docs then, so that users can find it easily.
>>
>> As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute a
>> new connector.
>>
>> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]> wrote:
>>
>> > Hi Stephan,
>> >
>> > I have no doubt about the value of adding Pulsar connector to Flink repo.
>> > My concern is about how exactly we are going to do it.
>> >
>> > As mentioned before, I believe that we can handle connectors more
>> > > pragmatically and less strict than the core of Flink, if it helps
>> > unlocking
>> > > users.
>> >
>> > I can see the benefit of being less restrict for the initial connector
>> code
>> > adoption. However, I don't think we should be less restrict on the
>> > maintenance commitment once the code is in Flink repo. It only makes
>> sense
>> > to check in something and ask users to use if we plan to maintain it.
>> >
>> > If I understand correctly, the current plan so far is following:
>> > 1. release 1.10
>> >    - Check in Pulsar connector on old interface and label it as beta
>> > version.
>> >    - encourage users to try it and report bugs.
>> > 2. release 1.11
>> >    - Check in Pulsar connector on new interface (a.k.a new Pulsar
>> > connector) and label it as beta version
>> >    - Deprecate the old Pulsar connector
>> >    - Fix bugs reported on old Pulsar connector from release 1.10
>> >    - Ask users to migrate from old Pulsar connector to new Pulsar
>> connector
>> > 3. release 1.12
>> >    - Announce end of support for old Pulsar connector and remove the code
>> >    - Fix bugs reported on new Pulsar connector.
>> >
>> > If this is the plan, it seems neither Flink nor the users trying the old
>> > Pulsar connector will benefit from this experimental old Pulsar
>> connector,
>> > because whatever feedbacks we got or bugs we fix on the old Pulsar
>> > connector are immediately thrown away in one or two releases.
>> >
>> > If we check in the old Pulsar connector right now, the only option I see
>> is
>> > to maintain it for a while (e.g. a year or more). IMO, the immediate
>> > deprecation and code removal hurts the users much more than asking them
>> to
>> > wait for another release. I personally think that we can avoid this
>> > maintenance burden by going directly to the new Pulsar connector,
>> > especially given that users can still use the connector even if they are
>> > not in Flink repo. That said, I am OK with maintaining both old and new
>> > Pulsar connector if we believe that having the Pulsar connector available
>> > right now in Flink repo is more important.
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]> wrote:
>> >
>> > > My assumption is as Sijie's, that once the connector is either part of
>> > > Flink, or part of the streamnative repo. No double maintenance.
>> > >
>> > > I feel this discussion is very much caught in problems that are all
>> > > solvable if we want to solve them.
>> > > Maybe we can think what our goal for users and the communities is?
>> > >
>> > >   - Do we want to help build a relationship between the Pulsar and
>> Flink
>> > > open source communities?
>> > >   - Will users find a connector in the streamnative repository?
>> > >   - Will users trust a connector that is not part of Flink as much?
>> > >
>> > > And then decide what is best according to the overall goals there.
>> > > As mentioned before, I believe that we can handle connectors more
>> > > pragmatically and less strict than the core of Flink, if it helps
>> > unlocking
>> > > users.
>> > >
>> > > Best,
>> > > Stephan
>> > >
>> > >
>> > >
>> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]> wrote:
>> > >
>> > > > Thanks Becket.
>> > > >
>> > > > I think it is better for the Flink community to judge the benefits of
>> > > doing
>> > > > this. I was trying to provide some views from outsiders.
>> > > >
>> > > > Thanks,
>> > > > Sijie
>> > > >
>> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]>
>> > > wrote:
>> > > >
>> > > > > Hi Sijie,
>> > > > >
>> > > > > Yes, we will have to support existing old connectors and new
>> > connectors
>> > > > in
>> > > > > parallel for a while. We have to take that maintenance overhead
>> > because
>> > > > > existing connectors have been used by the users for a long time. I
>> > > guess
>> > > > It
>> > > > > may take at least a year for us to fully remove the old connectors.
>> > > > >
>> > > > > Process wise, we can do the same for Pulsar connector. But I am not
>> > > sure
>> > > > if
>> > > > > we want to have the same burden on Pulsar connector, and I would
>> like
>> > > to
>> > > > > understand the benefit of doing that.
>> > > > >
>> > > > > For users, the benefit of having the old Pulsar connector checked
>> in
>> > > > seems
>> > > > > limited because 1) that code base will be immediately deprecated in
>> > the
>> > > > > next release in 3-4 months; 2) users can always use it even if it
>> is
>> > > not
>> > > > in
>> > > > > the Flink code base. Admittedly it is not as convenient as having
>> it
>> > in
>> > > > > Flink code base, but doesn't seem super either. And after 3-4
>> months,
>> > > > users
>> > > > > can just use the new connector in Flink repo.
>> > > > >
>> > > > > For Flink developers, the old connector code base is not something
>> > that
>> > > > we
>> > > > > want to evolve later. Instead, these code will be deprecated and
>> > > > > removed. So why do we want to get a beta version out to attract
>> > people
>> > > to
>> > > > > use something we don't want to maintain?
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jiangjie (Becket) Qin
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]>
>> > wrote:
>> > > > >
>> > > > > > Thanks everyone here. Sorry for jumping into the discussion here.
>> > > > > >
>> > > > > > I am not very familiar about the deprecation process in Flink.
>> If I
>> > > > > > misunderstood the process, please fix me.
>> > > > > >
>> > > > > > As far as I understand, FLIP-27 is introducing a new unified API
>> > for
>> > > > > > connectors. After it introduces the new API
>> > > > > > and before moving all the existing connectors from old API to new
>> > > API,
>> > > > > both
>> > > > > > old ApI and new API will co-exist
>> > > > > > for a while until Flink moves all existing connectors to new API.
>> > So
>> > > > the
>> > > > > > Pulsar connector (using old API) can
>> > > > > > follow the deprecation process with other connector using old API
>> > and
>> > > > the
>> > > > > > deprecation of old API, no?
>> > > > > >
>> > > > > > If that's the case, I think contributing the current connector
>> back
>> > > to
>> > > > > > Flink rather than maintaining it outside Flink
>> > > > > > would provide a bit more benefits. We can deprecate the existing
>> > > > > > streamnative/pulsar-flink repo and point the users
>> > > > > > to use the connector in Flink repo. So all the review processes
>> > will
>> > > > > happen
>> > > > > > within Flink for both old connector and
>> > > > > > new connector. It also reduces the confusions for the users as
>> the
>> > > > > > documentation and code base happen in one place.
>> > > > > >
>> > > > > > Thoughts?
>> > > > > >
>> > > > > > - Sijie
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <
>> [hidden email]>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Thanks for the explanation, Stephan. I have a few questions /
>> > > > thoughts.
>> > > > > > >
>> > > > > > > So that means we will remove the old connector without a major
>> > > > version
>> > > > > > > bump, is that correct?
>> > > > > > >
>> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11
>> connectors
>> > > > will
>> > > > > > > always work because we saw some dependency class collisions in
>> > the
>> > > > > past.
>> > > > > > To
>> > > > > > > make it safe we may have to maintain the old code for one more
>> > > > release.
>> > > > > > >
>> > > > > > > To be honest I am still wondering if we have to put the old
>> > > connector
>> > > > > in
>> > > > > > > Flink repo. if we check in the old connector to Flink. We will
>> > end
>> > > up
>> > > > > in
>> > > > > > > the following situation:
>> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
>> > > > > > > 2. Old connector in Flink Repo, which may be different from the
>> > one
>> > > > in
>> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in
>> 1.12)
>> > > > > > > 3. New connector in Flink Repo.
>> > > > > > >
>> > > > > > > We need to think about how to make the users in each case
>> happy.
>> > > > > > > - For users of (1), I assume Sijie and Yijie will have to
>> > maintain
>> > > > the
>> > > > > > code
>> > > > > > > a bit longer for its own compatibility even after we have (2).
>> In
>> > > > that
>> > > > > > > case, bugs found in old connector may or may not need to be
>> fixed
>> > > in
>> > > > > both
>> > > > > > > Flink and the streamnative/pulsar-flink repo.
>> > > > > > > - For users of (2), will we provide bug fixes? If we do, it
>> will
>> > > be a
>> > > > > > > little awkward because those bug fixes will be immediately
>> > > deprecated
>> > > > > in
>> > > > > > > 1.11, and removed in 1.12. So we are essentially asking users
>> to
>> > > > > migrate
>> > > > > > > away from the bug fix. After Flink 1.12, users may still have
>> to
>> > > > switch
>> > > > > > to
>> > > > > > > use (3) due to the potential dependency class conflicts
>> mentioned
>> > > > > above.
>> > > > > > > - Users of (3) have a much easier life and don't need to worry
>> > too
>> > > > > much.
>> > > > > > >
>> > > > > > > The above story seems a little complicated to tell. I think it
>> > will
>> > > > be
>> > > > > > much
>> > > > > > > easier to not have (2) at all.
>> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
>> > > > > > > 3. New connector in Flink Repo.
>> > > > > > >
>> > > > > > > - Old connector will only be maintained in
>> > > streamnative/pulsar-flink
>> > > > > repo
>> > > > > > > until it is fully deprecated. Users can always use the existing
>> > > > Pulsar
>> > > > > > > connector in that repo.
>> > > > > > > - New connector will be in Flink repo and maintained like the
>> > other
>> > > > > > > connectors.
>> > > > > > >
>> > > > > > > This seems much simpler for users understand and they will not
>> be
>> > > > > blocked
>> > > > > > > from using the old connector. If the concern is about the
>> quality
>> > > of
>> > > > > the
>> > > > > > > connector in streamnative/pulsar-flink repo, is it enough for
>> us
>> > > just
>> > > > > to
>> > > > > > > review the code in streamnative/pulsar-flink connector to make
>> > sure
>> > > > it
>> > > > > > > looks good from Flink's perspective?
>> > > > > > >
>> > > > > > > What do you think?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > >
>> > > > > > > Jiangjie (Becket) Qin
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <[hidden email]
>> >
>> > > > wrote:
>> > > > > > >
>> > > > > > > > My take would be the following:
>> > > > > > > >
>> > > > > > > >   - If we merge the connector now and replace it with a
>> FLIP-27
>> > > > > version
>> > > > > > > > before the 1.10 release, then we need no deprecation process
>> > > > > > > >   - If we don't manage to replace it with a FLIP-27 version
>> > > before
>> > > > > the
>> > > > > > > 1.10
>> > > > > > > > release, than it is good that we have the other version, so
>> no
>> > > > users
>> > > > > > get
>> > > > > > > > blocked.
>> > > > > > > >
>> > > > > > > > In the latter case we can see how we want to do it. Immediate
>> > > > removal
>> > > > > > of
>> > > > > > > > the old version or deprecation label and keeping it for one
>> > more
>> > > > > > release.
>> > > > > > > > Given that you should be able to use a Flink 1.10 connector
>> > with
>> > > > > Flink
>> > > > > > > 1.11
>> > > > > > > > as well (stable public APIs) there is also a workaround if
>> you
>> > > need
>> > > > > an
>> > > > > > > old
>> > > > > > > > connector in a newer version. So immediate removal might even
>> > be
>> > > > > > > feasible.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
>> > > [hidden email]>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Stephan,
>> > > > > > > > >
>> > > > > > > > > Thanks for the clarification. I completely agree with you
>> and
>> > > > > Thomas
>> > > > > > on
>> > > > > > > > the
>> > > > > > > > > process of adding connectors to Flink repo. However, I am
>> > > > wondering
>> > > > > > > what
>> > > > > > > > is
>> > > > > > > > > the deprecation process? Given the main concern here was
>> that
>> > > we
>> > > > > may
>> > > > > > > have
>> > > > > > > > > to maintain two Pulsar connector code bases until the old
>> one
>> > > is
>> > > > > > > removed
>> > > > > > > > > from the repo, it would be good to know how long we have to
>> > do
>> > > > > that.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > >
>> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
>> > [hidden email]
>> > > >
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Some quick thoughts on the connector contribution
>> process.
>> > I
>> > > > > > > basically
>> > > > > > > > > > reiterate here what Thomas mentioned in another thread
>> > about
>> > > > the
>> > > > > > > > Kinesis
>> > > > > > > > > > connector.
>> > > > > > > > > >
>> > > > > > > > > > For connectors, we should favor a low-overhead
>> contribution
>> > > > > > process,
>> > > > > > > > and
>> > > > > > > > > > accept user code and changes more readily than in the
>> core
>> > > > > system.
>> > > > > > > > > > That is because connectors have both a big variety of
>> > > scenarios
>> > > > > > they
>> > > > > > > > get
>> > > > > > > > > > used in (only through use and many small contributions do
>> > > they
>> > > > > > become
>> > > > > > > > > > really useful over time) and at the same time, and
>> > committers
>> > > > do
>> > > > > > not
>> > > > > > > > use
>> > > > > > > > > > the connector themselves and usually cannot foresee too
>> > well
>> > > > what
>> > > > > > is
>> > > > > > > > > > needed.
>> > > > > > > > > >
>> > > > > > > > > > Further more, a missing connector (or connector feature)
>> is
>> > > > > often a
>> > > > > > > > > bigger
>> > > > > > > > > > show stopper for users than a missing API or system
>> > feature.
>> > > > > > > > > >
>> > > > > > > > > > Along these lines of thougt, the conclusion would be to
>> > take
>> > > > the
>> > > > > > > Pulsar
>> > > > > > > > > > connector now, focus the review on
>> legal/dependencies/rough
>> > > > code
>> > > > > > > style
>> > > > > > > > > and
>> > > > > > > > > > conventions, label it as "beta" (in the sense of "new
>> code"
>> > > > that
>> > > > > is
>> > > > > > > > "not
>> > > > > > > > > > yet tested through longer use") and go ahead. And then
>> > evolve
>> > > > it
>> > > > > > > > quickly
>> > > > > > > > > > without putting formal blockers in the way, meaning also
>> > > > adding a
>> > > > > > new
>> > > > > > > > > FLIP
>> > > > > > > > > > 27 version when it is there.
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Stephan
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
>> > > > [hidden email]
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Yijie,
>> > > > > > > > > > >
>> > > > > > > > > > > Could you please follow the FLIP process to start a new
>> > > FLIP
>> > > > > > > > > [DISCUSSION]
>> > > > > > > > > > > thread in the mailing list?
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
>> > > > > > > > > > >
>> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list now.
>> So
>> > > > there
>> > > > > > is a
>> > > > > > > > > FLIP
>> > > > > > > > > > > number collision. Can you change the FLIP number to 72?
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > >
>> > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
>> > > > > [hidden email]>
>> > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi Yijie,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks for sharing the pulsar FLIP.
>> > > > > > > > > > > > Would you mind enabling comments/suggestions on the
>> > > google
>> > > > > doc
>> > > > > > > > link?
>> > > > > > > > > > This
>> > > > > > > > > > > > way the contributors from the community can comment
>> on
>> > > the
>> > > > > doc.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best,
>> > > > > > > > > > > > Rong
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
>> > > > > > > > > [hidden email]>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hello everyone,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I've drafted a FLIP that describes the current
>> design
>> > > of
>> > > > > the
>> > > > > > > > Pulsar
>> > > > > > > > > > > > > connector:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Please take a look and let me know what you think.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > Yijie
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
>> > > > > > > [hidden email]>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi All,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Sorry for joining the discussion late and thanks
>> > > Yijie
>> > > > &
>> > > > > > > Sijie
>> > > > > > > > > for
>> > > > > > > > > > > > > driving
>> > > > > > > > > > > > > > the discussion.
>> > > > > > > > > > > > > > I also think the Pulsar connector would be a very
>> > > > > valuable
>> > > > > > > > > addition
>> > > > > > > > > > > to
>> > > > > > > > > > > > > > Flink. I can also help out a bit on the review
>> side
>> > > :-)
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Regarding the timeline, I also share concerns
>> with
>> > > > Becket
>> > > > > > on
>> > > > > > > > the
>> > > > > > > > > > > > > > relationship between the new Pulsar connector and
>> > > > > FLIP-27.
>> > > > > > > > > > > > > > There's also another discussion just started by
>> > > Stephan
>> > > > > on
>> > > > > > > > > dropping
>> > > > > > > > > > > > Kafka
>> > > > > > > > > > > > > > 9/10 support on next Flink release [1].  Although
>> > the
>> > > > > > > situation
>> > > > > > > > > is
>> > > > > > > > > > > > > somewhat
>> > > > > > > > > > > > > > different, and Kafka 9/10 connector has been in
>> > Flink
>> > > > for
>> > > > > > > > almost
>> > > > > > > > > > 3-4
>> > > > > > > > > > > > > years,
>> > > > > > > > > > > > > > based on the discussion I am not sure if a major
>> > > > version
>> > > > > > > > release
>> > > > > > > > > > is a
>> > > > > > > > > > > > > > requirement for removing old connector supports.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I think there shouldn't be a blocker if we agree
>> > the
>> > > > old
>> > > > > > > > > connector
>> > > > > > > > > > > will
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is
>> > there.
>> > > > As
>> > > > > > > > Stephan
>> > > > > > > > > > > > stated,
>> > > > > > > > > > > > > it
>> > > > > > > > > > > > > > is easier to contribute the source sooner and
>> > adjust
>> > > it
>> > > > > > > later.
>> > > > > > > > > > > > > > We should also ensure we clearly communicate the
>> > > > message:
>> > > > > > for
>> > > > > > > > > > > example,
>> > > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27
>> > > > connector
>> > > > > > page
>> > > > > > > > of
>> > > > > > > > > > the
>> > > > > > > > > > > > > > website, documentations, etc. Any other thoughts?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > Rong
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > [1]
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
>> > > > > > > > [hidden email]
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Technically speaking, removing the old
>> connector
>> > > code
>> > > > > is
>> > > > > > a
>> > > > > > > > > > > backwards
>> > > > > > > > > > > > > > > incompatible change which requires a major
>> > version
>> > > > > bump,
>> > > > > > > i.e.
>> > > > > > > > > > Flink
>> > > > > > > > > > > > > 2.x.
>> > > > > > > > > > > > > > > Given that we don't have a clear plan on when
>> to
>> > > have
>> > > > > the
>> > > > > > > > next
>> > > > > > > > > > > major
>> > > > > > > > > > > > > > > version release, it seems unclear how long the
>> > old
>> > > > > > > connector
>> > > > > > > > > code
>> > > > > > > > > > > > will
>> > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > there if we check it in right now. Or will we
>> > > remove
>> > > > > the
>> > > > > > > old
>> > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > without a major version bump? In any case, it
>> > > sounds
>> > > > > not
>> > > > > > > > quite
>> > > > > > > > > > user
>> > > > > > > > > > > > > > > friendly to the those who might use the old
>> > Pulsar
>> > > > > > > > connector. I
>> > > > > > > > > > am
>> > > > > > > > > > > > not
>> > > > > > > > > > > > > sure
>> > > > > > > > > > > > > > > if it is worth these potential problems in
>> order
>> > to
>> > > > > have
>> > > > > > > the
>> > > > > > > > > > Pulsar
>> > > > > > > > > > > > > source
>> > > > > > > > > > > > > > > connector checked in one or two months earlier.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan Ewen <
>> > > > > > > > [hidden email]
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Agreed, if we check in the old code, we
>> should
>> > > make
>> > > > > it
>> > > > > > > > clear
>> > > > > > > > > > that
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based
>> version
>> > > of
>> > > > > the
>> > > > > > > > > > connector
>> > > > > > > > > > > is
>> > > > > > > > > > > > > > > there.
>> > > > > > > > > > > > > > > > We should not commit to maintaining the old
>> > > > version,
>> > > > > > that
>> > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > > > indeed
>> > > > > > > > > > > > > > > > too much overhead.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket Qin <
>> > > > > > > > > > [hidden email]
>> > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Hi Stephan,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Thanks for the volunteering to help.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Yes, the overhead would just be review
>> > > capacity.
>> > > > In
>> > > > > > > > fact, I
>> > > > > > > > > > am
>> > > > > > > > > > > > not
>> > > > > > > > > > > > > > > > worrying
>> > > > > > > > > > > > > > > > > too much about the review capacity. That is
>> > > just
>> > > > a
>> > > > > > one
>> > > > > > > > time
>> > > > > > > > > > > cost.
>> > > > > > > > > > > > > My
>> > > > > > > > > > > > > > > > > concern is mainly about the long term
>> burden.
>> > > > > Assume
>> > > > > > we
>> > > > > > > > > have
>> > > > > > > > > > > new
>> > > > > > > > > > > > > source
>> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly added
>> > Pulsar
>> > > > > > > > connectors
>> > > > > > > > > in
>> > > > > > > > > > > old
>> > > > > > > > > > > > > > > > > interface. Later on if we migrate Pulsar to
>> > new
>> > > > > > source
>> > > > > > > > > > > interface,
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > old
>> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated almost
>> > > > > > immediately
>> > > > > > > > > after
>> > > > > > > > > > > > > checked
>> > > > > > > > > > > > > > > in,
>> > > > > > > > > > > > > > > > > but we may still have to maintain two code
>> > > bases.
>> > > > > For
>> > > > > > > the
>> > > > > > > > > > > > existing
>> > > > > > > > > > > > > > > > > connectors, we have to do that anyways. But
>> > it
>> > > > > would
>> > > > > > be
>> > > > > > > > > good
>> > > > > > > > > > to
>> > > > > > > > > > > > > avoid
>> > > > > > > > > > > > > > > > > introducing a new connector with the same
>> > > > problem.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan
>> Ewen
>> > <
>> > > > > > > > > > [hidden email]
>> > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Hi all!
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Nice to see this lively discussion about
>> > the
>> > > > > Pulsar
>> > > > > > > > > > > connector.
>> > > > > > > > > > > > > > > > > > Some thoughts on the open questions:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as a
>> > > > community
>> > > > > > > > package
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Looks like the discussion is more going
>> > > towards
>> > > > > > > > > > > contribution. I
>> > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > is good, especially if we think that we
>> > want
>> > > to
>> > > > > > > build a
>> > > > > > > > > > > > similarly
>> > > > > > > > > > > > > > > deep
>> > > > > > > > > > > > > > > > > > integration with Pulsar as we have for
>> > > example
>> > > > > with
>> > > > > > > > > Kafka.
>> > > > > > > > > > > The
>> > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > already looks like a more thorough
>> > connector
>> > > > than
>> > > > > > > many
>> > > > > > > > > > others
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > the repository.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > With either a repo split, or the new
>> build
>> > > > > system,
>> > > > > > I
>> > > > > > > > hope
>> > > > > > > > > > > that
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > build
>> > > > > > > > > > > > > > > > > > overhead is not a problem.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > ## Committer Support
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Becket offered some help already, I can
>> > also
>> > > > > help a
>> > > > > > > > bit.
>> > > > > > > > > I
>> > > > > > > > > > > hope
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > between us, we can cover this.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually
>> making
>> > > some
>> > > > > > > PoC-ing
>> > > > > > > > > > > > > progress, but
>> > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > take 2 more months, I would estimate,
>> > before
>> > > it
>> > > > > is
>> > > > > > > > fully
>> > > > > > > > > > > > > available.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > If we want to be on the safe side with
>> the
>> > > > > > > > contribution,
>> > > > > > > > > we
>> > > > > > > > > > > > > should
>> > > > > > > > > > > > > > > > > > contribute the source sooner and adjust
>> it
>> > > > later.
>> > > > > > > That
>> > > > > > > > > > would
>> > > > > > > > > > > > also
>> > > > > > > > > > > > > > > help
>> > > > > > > > > > > > > > > > us
>> > > > > > > > > > > > > > > > > > in case things get crazy towards the 1.10
>> > > > feature
>> > > > > > > > freeze
>> > > > > > > > > > and
>> > > > > > > > > > > it
>> > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > hard to find time to review the new
>> > changes.
>> > > > > > > > > > > > > > > > > > What would be the overhead of
>> contributing
>> > > now?
>> > > > > > Given
>> > > > > > > > > that
>> > > > > > > > > > > the
>> > > > > > > > > > > > > code
>> > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > already there, it looks like it would be
>> > only
>> > > > > > review
>> > > > > > > > > > > capacity,
>> > > > > > > > > > > > > right?
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > Stephan
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie
>> > Shen <
>> > > > > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Hi everyone!
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Thanks for your attention and the
>> > promotion
>> > > > of
>> > > > > > this
>> > > > > > > > > work.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as
>> > possible
>> > > > for
>> > > > > > more
>> > > > > > > > > > > specific
>> > > > > > > > > > > > > > > > > discussions.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have not
>> > > > reached
>> > > > > a
>> > > > > > > > > > consensus.
>> > > > > > > > > > > > > > > > Therefore,
>> > > > > > > > > > > > > > > > > > > I will explain all the functionalities
>> of
>> > > the
>> > > > > > > > existing
>> > > > > > > > > > > > > connector in
>> > > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, and
>> > > > Catalog)
>> > > > > to
>> > > > > > > > > > continue
>> > > > > > > > > > > > our
>> > > > > > > > > > > > > > > > > > > discussions in FLIP.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Thanks for your kind help.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > Yijie
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM Becket
>> > Qin
>> > > <
>> > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Hi Sijie,
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > If we agree that the goal is to have
>> > > Pulsar
>> > > > > > > > connector
>> > > > > > > > > > in
>> > > > > > > > > > > > > 1.10,
>> > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > do the following:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar
>> connector
>> > > to
>> > > > > > Flink
>> > > > > > > > main
>> > > > > > > > > > > repo
>> > > > > > > > > > > > > as it
>> > > > > > > > > > > > > > > > is a
>> > > > > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > > public interface to Flink main repo.
>> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink
>> > right
>> > > > away
>> > > > > > as
>> > > > > > > > > there
>> > > > > > > > > > is
>> > > > > > > > > > > > no
>> > > > > > > > > > > > > > > change
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > sink interface so far.
>> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27.
>> Flink
>> > > 1.10
>> > > > > is
>> > > > > > > > going
>> > > > > > > > > to
>> > > > > > > > > > > be
>> > > > > > > > > > > > > code
>> > > > > > > > > > > > > > > > > freeze
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > late Nov and let's say we give a
>> month
>> > to
>> > > > the
>> > > > > > > > > > development
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > review
>> > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have
>> > FLIP-27
>> > > > by
>> > > > > > late
>> > > > > > > > > Oct.
>> > > > > > > > > > > > There
>> > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > still
>> > > > > > > > > > > > > > > > > > 7
>> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it is
>> doable.
>> > > If
>> > > > > > > FLIP-27
>> > > > > > > > is
>> > > > > > > > > > not
>> > > > > > > > > > > > > ready
>> > > > > > > > > > > > > > > by
>> > > > > > > > > > > > > > > > > late
>> > > > > > > > > > > > > > > > > > > > Oct, we can review and check in
>> Pulsar
>> > > > > > connector
>> > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > existing
>> > > > > > > > > > > > > > > > > > source
>> > > > > > > > > > > > > > > > > > > > interface. This means we will have
>> > Pulsar
>> > > > > > > connector
>> > > > > > > > > in
>> > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > 1.10,
>> > > > > > > > > > > > > > > > > > either
>> > > > > > > > > > > > > > > > > > > > with or without FLIP-27.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Because we are going to have Pulsar
>> > sink
>> > > > and
>> > > > > > > source
>> > > > > > > > > > > checked
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > separately,
>> > > > > > > > > > > > > > > > > > > > it might make sense to have two
>> FLIPs,
>> > > one
>> > > > > for
>> > > > > > > > Pulsar
>> > > > > > > > > > > sink
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > another
>> > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can start the
>> > work
>> > > on
>> > > > > > > Pulsar
>> > > > > > > > > sink
>> > > > > > > > > > > > right
>> > > > > > > > > > > > > > > away.
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM Sijie
>> > Guo
>> > > <
>> > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > What's the take from Flink
>> community?
>> > > > Shall
>> > > > > > we
>> > > > > > > > wait
>> > > > > > > > > > for
>> > > > > > > > > > > > > FLIP-27
>> > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > shall we
>> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And what the
>> > > next
>> > > > > > steps
>> > > > > > > > are?
>> > > > > > > > > > :-)
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > > > Sijie
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM
>> Bowen
>> > > Li <
>> > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Hi,
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar connector
>> > in
>> > > > > Flink
>> > > > > > > can
>> > > > > > > > > be a
>> > > > > > > > > > > > good
>> > > > > > > > > > > > > > > mutual
>> > > > > > > > > > > > > > > > > > > benefit
>> > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > both communities.
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Another perspective is that
>> Pulsar
>> > > > > > connector
>> > > > > > > is
>> > > > > > > > > the
>> > > > > > > > > > > 1st
>> > > > > > > > > > > > > > > > streaming
>> > > > > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > > > > that integrates with Flink's
>> > metadata
>> > > > > > > > management
>> > > > > > > > > > > system
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > Catalog
>> > > > > > > > > > > > > > > > > > > APIs.
>> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the
>> > > > integration
>> > > > > > > turns
>> > > > > > > > > out
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > > whether
>> > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > need to
>> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack,
>> which
>> > > are
>> > > > > > > > currently
>> > > > > > > > > in
>> > > > > > > > > > > > > Beta, to
>> > > > > > > > > > > > > > > > > cater
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus I'm
>> in
>> > > > favor
>> > > > > of
>> > > > > > > > > merging
>> > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > into
>> > > > > > > > > > > > > > > > > > > > > > Flink 1.10.
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller
>> sized
>> > > > PRs,
>> > > > > > e.g.
>> > > > > > > > > maybe
>> > > > > > > > > > > one
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > basic
>> > > > > > > > > > > > > > > > > > > > > > source/sink functionalities and
>> > > another
>> > > > > for
>> > > > > > > > > schema
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > > > > > > > integration,
>> > > > > > > > > > > > > > > > > > > > > > just to make them easier to
>> review.
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to wait
>> for
>> > > > > > FLIP-27.
>> > > > > > > > But
>> > > > > > > > > I
>> > > > > > > > > > > > don't
>> > > > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > > > FLIP-27
>> > > > > > > > > > > > > > > > > > > > > > should be a blocker in cases
>> where
>> > it
>> > > > > > cannot
>> > > > > > > > make
>> > > > > > > > > > its
>> > > > > > > > > > > > way
>> > > > > > > > > > > > > > > into
>> > > > > > > > > > > > > > > > > 1.10
>> > > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable amount
>> of
>> > > time
>> > > > > for
>> > > > > > > > > > > committers
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > review
>> > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully adapt
>> to
>> > > new
>> > > > > > > > > interfaces.
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Bowen
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM
>> > Becket
>> > > > > Qin <
>> > > > > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Hi Till,
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > You are right. It all depends
>> on
>> > > when
>> > > > > the
>> > > > > > > new
>> > > > > > > > > > > source
>> > > > > > > > > > > > > > > > interface
>> > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > going
>> > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I think it
>> > > would
>> > > > > be
>> > > > > > > > there
>> > > > > > > > > in
>> > > > > > > > > > > > > about a
>> > > > > > > > > > > > > > > > month
>> > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > so.
>> > > > > > > > > > > > > > > > > > > > > > But
>> > > > > > > > > > > > > > > > > > > > > > > I could be too optimistic. It
>> > would
>> > > > > also
>> > > > > > be
>> > > > > > > > > good
>> > > > > > > > > > to
>> > > > > > > > > > > > > hear
>> > > > > > > > > > > > > > > what
>> > > > > > > > > > > > > > > > > do
>> > > > > > > > > > > > > > > > > > > > > Aljoscha
>> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as they are
>> > also
>> > > > > > involved
>> > > > > > > > in
>> > > > > > > > > > > > FLIP-27.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > In general I think we should
>> have
>> > > > > Pulsar
>> > > > > > > > > > connector
>> > > > > > > > > > > in
>> > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > 1.10,
>> > > > > > > > > > > > > > > > > > > > > > > preferably with the new source
>> > > > > interface.
>> > > > > > > We
>> > > > > > > > > can
>> > > > > > > > > > > also
>> > > > > > > > > > > > > check
>> > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > right
>> > > > > > > > > > > > > > > > > > > > > > now
>> > > > > > > > > > > > > > > > > > > > > > > with old source interface, but
>> I
>> > > > > suspect
>> > > > > > > few
>> > > > > > > > > > users
>> > > > > > > > > > > > > will use
>> > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > before
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > next official release.
>> Therefore,
>> > > it
>> > > > > > seems
>> > > > > > > > > > > reasonable
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > wait a
>> > > > > > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > > > > > bit
>> > > > > > > > > > > > > > > > > > > > > > > to see whether we can jump to
>> the
>> > > new
>> > > > > > > source
>> > > > > > > > > > > > > interface. As
>> > > > > > > > > > > > > > > > long
>> > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > > make
>> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it,
>> waiting a
>> > > > > little
>> > > > > > > bit
>> > > > > > > > > > > doesn't
>> > > > > > > > > > > > > seem
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > hurt
>> > > > > > > > > > > > > > > > > > > much.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 PM
>> > Till
>> > > > > > > Rohrmann
>> > > > > > > > <
>> > > > > > > > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the
>> problem
>> > > > would
>> > > > > be
>> > > > > > > if
>> > > > > > > > we
>> > > > > > > > > > > > > committed
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > connector before the new
>> source
>> > > > > > interface
>> > > > > > > > is
>> > > > > > > > > > > ready.
>> > > > > > > > > > > > > If I
>> > > > > > > > > > > > > > > > > > > understood
>> > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to
>> > > support
>> > > > > the
>> > > > > > > old
>> > > > > > > > > > source
>> > > > > > > > > > > > > > > interface
>> > > > > > > > > > > > > > > > > > > anyway
>> > > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > existing connectors. By
>> > checking
>> > > it
>> > > > > in
>> > > > > > > > early
>> > > > > > > > > I
>> > > > > > > > > > > > could
>> > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > benefit
>> > > > > > > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > > > > > > > our users could start using
>> the
>> > > > > > connector
>> > > > > > > > > > > earlier.
>> > > > > > > > > > > > > > > > Moreover,
>> > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar
>> > > integration
>> > > > > is
>> > > > > > > > being
>> > > > > > > > > > > > delayed
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > case
>> > > > > > > > > > > > > > > > > > > that the
>> > > > > > > > > > > > > > > > > > > > > > > > source interface should be
>> > > delayed.
>> > > > > The
>> > > > > > > > only
>> > > > > > > > > > > > > downside I
>> > > > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > extra
>> > > > > > > > > > > > > > > > > > > > > > > > review effort and potential
>> > fixes
>> > > > > which
>> > > > > > > > might
>> > > > > > > > > > be
>> > > > > > > > > > > > > > > irrelevant
>> > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > > > > > > source interface
>> > implementation.
>> > > I
>> > > > > > guess
>> > > > > > > it
>> > > > > > > > > > > mainly
>> > > > > > > > > > > > > > > depends
>> > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > > > > > > certain
>> > > > > > > > > > > > > > > > > > > > > > > > we are when the new source
>> > > > interface
>> > > > > > will
>> > > > > > > > be
>> > > > > > > > > > > ready.
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > > > > > > > > > > > > Till
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:56
>> AM
>> > > > Becket
>> > > > > > > Qin <
>> > > > > > > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your
>> > > thoughts.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some
>> update
>> > > on
>> > > > > > > FLIP-27.
>> > > > > > > > > > > > Although
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > > > > wiki
>> > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread has been
>> > > quiet
>> > > > > for
>> > > > > > > some
>> > > > > > > > > > > time, a
>> > > > > > > > > > > > > few
>> > > > > > > > > > > > > > > > > > > committer /
>> > > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink
>> > community
>> > > > > were
>> > > > > > > > > actually
>> > > > > > > > > > > > > > > prototyping
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > entire
>> > > > > > > > > > > > > > > > > > > > > > > > thing.
>> > > > > > > > > > > > > > > > > > > > > > > > > We have made some good
>> > progress
>> > > > > there
>> > > > > > > but
>> > > > > > > > > > want
>> > > > > > > > > > > to
>> > > > > > > > > > > > > > > update
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > FLIP
>> > > > > > > > > > > > > > > > > > > > > > wiki
>> > > > > > > > > > > > > > > > > > > > > > > > > after the entire thing is
>> > > > verified
>> > > > > to
>> > > > > > > > work
>> > > > > > > > > in
>> > > > > > > > > > > > case
>> > > > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > > > > > > > last
>> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the
>> > > > > > implementation.
>> > > > > > > I
>> > > > > > > > > > don't
>> > > > > > > > > > > > > have an
>> > > > > > > > > > > > > > > > > exact
>> > > > > > > > > > > > > > > > > > > ETA
>> > > > > > > > > > > > > > > > > > > > > yet,
>> > > > > > > > > > > > > > > > > > > > > > > > but I
>> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be
>> > within
>> > > a
>> > > > > > month
>> > > > > > > or
>> > > > > > > > > so.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the
>> > > current
>> > > > > > Flink
>> > > > > > > > > Pulsar
>> > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > see if
>> > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It would be
>> > > good
>> > > > to
>> > > > > > > avoid
>> > > > > > > > > the
>> > > > > > > > > > > > case
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > checked
>> > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with some
>> > > review
>> > > > > > > efforts
>> > > > > > > > > and
>> > > > > > > > > > > > > shortly
>> > > > > > > > > > > > > > > > after
>> > > > > > > > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is ready.
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 8:39
>> > AM
>> > > > > Yijie
>> > > > > > > > Shen <
>> > > > > > > > > > > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the
>> feedback
>> > > and
>> > > > > > > > > > suggestions!
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the goal
>> of
>> > > the
>> > > > > > > > connector
>> > > > > > > > > > has
>> > > > > > > > > > > > > always
>> > > > > > > > > > > > > > > > been
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > provide
>> > > > > > > > > > > > > > > > > > > > > > > > > > users with the latest
>> > > features
>> > > > of
>> > > > > > > both
>> > > > > > > > > > > systems
>> > > > > > > > > > > > as
>> > > > > > > > > > > > > > > soon
>> > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > possible.
>> > > > > > > > > > > > > > > > > > > > > > > We
>> > > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute the
>> > > > > connector
>> > > > > > > to
>> > > > > > > > > > Flink
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > hope
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > get
>> > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and feedback
>> > from
>> > > > > Flink
>> > > > > > > > > experts
>> > > > > > > > > > > to
>> > > > > > > > > > > > > ensure
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > high
>> > > > > > > > > > > > > > > > > > > > > > > quality
>> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we noticed
>> its
>> > > > > > existence
>> > > > > > > > at
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > beginning
>> > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > reworking
>> > > > > > > > > > > > > > > > > > > > > > > > > > the connector
>> > implementation
>> > > > > based
>> > > > > > on
>> > > > > > > > > Flink
>> > > > > > > > > > > > 1.9;
>> > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > also
>> > > > > > > > > > > > > > > > > > > wanted
>> > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector that
>> > > supports
>> > > > > > both
>> > > > > > > > > batch
>> > > > > > > > > > > and
>> > > > > > > > > > > > > stream
>> > > > > > > > > > > > > > > > > > > computing
>> > > > > > > > > > > > > > > > > > > > > > based
>> > > > > > > > > > > > > > > > > > > > > > > > > > on it.
>> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has been
>> > inactive
>> > > > for
>> > > > > > > some
>> > > > > > > > > > time,
>> > > > > > > > > > > so
>> > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > decided
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > provide
>> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with most of
>> > the
>> > > > new
>> > > > > > > > > features,
>> > > > > > > > > > > such
>> > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > type
>> > > > > > > > > > > > > > > > > > > > > > > system
>> > > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog API
>> > > first.
>> > > > We
>> > > > > > > will
>> > > > > > > > > pay
>> > > > > > > > > > > > > attention
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > progress
>> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually
>> and
>> > > > > > > incorporate
>> > > > > > > > it
>> > > > > > > > > > > with
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > > soon
>> > > > > > > > > > > > > > > > > > > > > > > > > > as possible.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test status
>> > of
>> > > > the
>> > > > > > > > > connector,
>> > > > > > > > > > > we
>> > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > following
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > other
>> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in Flink
>> > > > > > repository
>> > > > > > > > and
>> > > > > > > > > > > aimed
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > provide
>> > > > > > > > > > > > > > > > > > > > > > throughout
>> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We are
>> > > also
>> > > > > > happy
>> > > > > > > to
>> > > > > > > > > > hear
>> > > > > > > > > > > > > > > > suggestions
>> > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from the
>> Flink
>> > > > > > community
>> > > > > > > to
>> > > > > > > > > > > improve
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > stability and
>> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the
>> > connector
>> > > > > > > > > continuously.
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
>> 5:59
>> > > AM
>> > > > > > Sijie
>> > > > > > > > Guo
>> > > > > > > > > <
>> > > > > > > > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for the
>> > > > > comments
>> > > > > > > and
>> > > > > > > > > > > > feedback.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that the
>> > > main
>> > > > > > > question
>> > > > > > > > > > here
>> > > > > > > > > > > is
>> > > > > > > > > > > > > > > about -
>> > > > > > > > > > > > > > > > > > "how
>> > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > community maintain the
>> > > > > > connector?".
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two thoughts
>> > from
>> > > > > > myself.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and
>> where
>> > to
>> > > > > host
>> > > > > > > this
>> > > > > > > > > > > > > integration
>> > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > kind
>> > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > less
>> > > > > > > > > > > > > > > > > > > > > > > > > > important
>> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe there
>> can
>> > > be
>> > > > > many
>> > > > > > > > ways
>> > > > > > > > > to
>> > > > > > > > > > > > > achieve
>> > > > > > > > > > > > > > > it.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the
>> > > contribution,
>> > > > > what
>> > > > > > > we
>> > > > > > > > > are
>> > > > > > > > > > > > > looking
>> > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > here
>> > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > how
>> > > > > > > > > > > > > > > > > > > > > > > > these
>> > > > > > > > > > > > > > > > > > > > > > > > > > two
>> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can build
>> the
>> > > > > > > > collaboration
>> > > > > > > > > > > > > > > relationship
>> > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > > > developing
>> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration between
>> > > > Pulsar
>> > > > > > and
>> > > > > > > > > Flink.
>> > > > > > > > > > > > Even
>> > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > try
>> > > > > > > > > > > > > > > > > > > our
>> > > > > > > > > > > > > > > > > > > > > > best
>> > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > catch
>> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates in
>> > Flink
>> > > > > > > > community.
>> > > > > > > > > We
>> > > > > > > > > > > are
>> > > > > > > > > > > > > still
>> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact that we
>> > > have
>> > > > > less
>> > > > > > > > > > > experiences
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > than
>> > > > > > > > > > > > > > > > > > > > > folks
>> > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In order to
>> > make
>> > > > > sure
>> > > > > > we
>> > > > > > > > > > > maintain
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > deliver
>> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality
>> > pulsar-flink
>> > > > > > > > integration
>> > > > > > > > > > to
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > users
>> > > > > > > > > > > > > > > > > who
>> > > > > > > > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we need
>> > some
>> > > > help
>> > > > > > > from
>> > > > > > > > > the
>> > > > > > > > > > > > > experts
>> > > > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > community.
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been
>> following
>> > > > > FLIP-27
>> > > > > > > > for a
>> > > > > > > > > > > > while.
>> > > > > > > > > > > > > > > > > Originally
>> > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > were
>> > > > > > > > > > > > > > > > > > > > > > > > > > thinking
>> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the
>> > > > connectors
>> > > > > > back
>> > > > > > > > > after
>> > > > > > > > > > > > > > > integrating
>> > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced in
>> > > > FLIP-27.
>> > > > > > But
>> > > > > > > we
>> > > > > > > > > > > decided
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > initiate
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > conversation
>> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as possible.
>> > > Because
>> > > > > we
>> > > > > > > > > believe
>> > > > > > > > > > > > there
>> > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > benefits
>> > > > > > > > > > > > > > > > > > > > > > > > doing
>> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than
>> later.
>> > > As
>> > > > > part
>> > > > > > > of
>> > > > > > > > > > > > > contribution,
>> > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > help
>> > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > community understand
>> more
>> > > > about
>> > > > > > > > Pulsar
>> > > > > > > > > > and
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > potential
>> > > > > > > > > > > > > > > > > > > > > > > integration
>> > > > > > > > > > > > > > > > > > > > > > > > > > points.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also help
>> > Flink
>> > > > > > > community
>> > > > > > > > > > > verify
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > > > > API
>> > > > > > > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > > > > > > well
>> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API (e.g.
>> > > > catalog
>> > > > > > > API).
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 at
>> > 5:24
>> > > > AM
>> > > > > > > Becket
>> > > > > > > > > > Qin <
>> > > > > > > > > > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the
>> interest
>> > > in
>> > > > > > > > > contributing
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > connector.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I think
>> > > having
>> > > > > > Pulsar
>> > > > > > > > > > > connector
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > strong
>> > > > > > > > > > > > > > > > > > > > > support
>> > > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition to
>> > > Flink.
>> > > > > So
>> > > > > > I
>> > > > > > > am
>> > > > > > > > > > happy
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > shepherd
>> > > > > > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > > > > > effort.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would
>> also
>> > > > like
>> > > > > to
>> > > > > > > > > provide
>> > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > context
>> > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > recent
>> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink connectors
>> > > > > ecosystem.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way Flink
>> > > > > maintains
>> > > > > > > its
>> > > > > > > > > > > > > connector has
>> > > > > > > > > > > > > > > > hit
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > scalability
>> > > > > > > > > > > > > > > > > > > > > > > > > > bar.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more
>> > > > connectors
>> > > > > > > > coming
>> > > > > > > > > > into
>> > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > repo,
>> > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > > > > > > facing a
>> > > > > > > > > > > > > > > > > > > > > > > > > > few
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as long
>> > > build
>> > > > > and
>> > > > > > > > > testing
>> > > > > > > > > > > > > time. To
>> > > > > > > > > > > > > > > > > > address
>> > > > > > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > > > > > > > > problem,
>> > > > > > > > > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to do
>> > the
>> > > > > > > following:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the
>> > > connectors
>> > > > > > into
>> > > > > > > a
>> > > > > > > > > > > separate
>> > > > > > > > > > > > > > > > > repository.
>> > > > > > > > > > > > > > > > > > > This
>> > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to
>> > potential
>> > > > > > solution
>> > > > > > > > to
>> > > > > > > > > > > > shorten
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > build
>> > > > > > > > > > > > > > > > > > > time.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the
>> > > connectors
>> > > > > to
>> > > > > > > stay
>> > > > > > > > > as
>> > > > > > > > > > > > > ecosystem
>> > > > > > > > > > > > > > > > > > project
>> > > > > > > > > > > > > > > > > > > > > while
>> > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > tries
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good
>> support
>> > > for
>> > > > > > > > > > functionality
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > compatibility
>> > > > > > > > > > > > > > > > > > > > > > > tests.
>> > > > > > > > > > > > > > > > > > > > > > > > > > Robert
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to create
>> a
>> > > > Flink
>> > > > > > > > > Ecosystem
>> > > > > > > > > > > > > project
>> > > > > > > > > > > > > > > > > website
>> > > > > > > > > > > > > > > > > > > and it
>> > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > going
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some final
>> > > approval
>> > > > > > > > process.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above
>> > efforts,
>> > > it
>> > > > > > would
>> > > > > > > > be
>> > > > > > > > > > > great
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > first
>> > > > > > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > > > > > if we
>> > > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector as
>> an
>> > > > > > ecosystem
>> > > > > > > > > > project
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > > great
>> > > > > > > > > > > > > > > > > > > support.
>> > > > > > > > > > > > > > > > > > > > > It
>> > > > > > > > > > > > > > > > > > > > > > > > would
>> > > > > > > > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how the
>> > > Flink
>> > > > > > Pulsar
>> > > > > > > > > > > connector
>> > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > tested
>> > > > > > > > > > > > > > > > > > > > > currently
>> > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > > > > > > > > > > > > if
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn
>> something
>> > to
>> > > > > > > maintain
>> > > > > > > > it
>> > > > > > > > > > as
>> > > > > > > > > > > an
>> > > > > > > > > > > > > > > > ecosystem
>> > > > > > > > > > > > > > > > > > > project
>> > > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > > > good
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test
>> > > coverage.
>> > > > If
>> > > > > > the
>> > > > > > > > > > quality
>> > > > > > > > > > > > as
>> > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > ecosystem
>> > > > > > > > > > > > > > > > > > > > > > project
>> > > > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > hard
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we may
>> as
>> > > > well
>> > > > > > > adopt
>> > > > > > > > it
>> > > > > > > > > > > into
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > main
>> > > > > > > > > > > > > > > > > > repo.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another ongoing
>> > > effort
>> > > > > is
>> > > > > > > > > FLIP-27
>> > > > > > > > > > > > where
>> > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > making
>> > > > > > > > > > > > > > > > > > > > > > > changes
>> > > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source
>> connector
>> > > > > > > architecture
>> > > > > > > > > and
>> > > > > > > > > > > > > > > interface.
>> > > > > > > > > > > > > > > > > This
>> > > > > > > > > > > > > > > > > > > > > change
>> > > > > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > > > > > > likely
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10.
>> Therefore
>> > > > > timing
>> > > > > > > > wise,
>> > > > > > > > > if
>> > > > > > > > > > > we
>> > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > going
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main
>> > repo, I
>> > > > am
>> > > > > > > > > wondering
>> > > > > > > > > > if
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > > > should
>> > > > > > > > > > > > > > > > > > hold
>> > > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > > little
>> > > > > > > > > > > > > > > > > > > > > > > > bit
>> > > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar
>> > connector
>> > > > > adapt
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > > new
>> > > > > > > > > > > > > > > interface
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > avoid
>> > > > > > > > > > > > > > > > > > > > > > > > shortly
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019
>> at
>> > > 4:32
>> > > > > PM
>> > > > > > > > > Chesnay
>> > > > > > > > > > > > > Schepler <
>> > > > > > > > > > > > > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite worried
>> > that
>> > > we
>> > > > > may
>> > > > > > > end
>> > > > > > > > > up
>> > > > > > > > > > > > > repeating
>> > > > > > > > > > > > > > > > > > history.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were already
>> 2
>> > > > > attempts
>> > > > > > > at
>> > > > > > > > > > > > > contributing a
>> > > > > > > > > > > > > > > > > > pulsar
>> > > > > > > > > > > > > > > > > > > > > > > connector,
>> > > > > > > > > > > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed
>> > because
>> > > > no
>> > > > > > > > > committer
>> > > > > > > > > > > was
>> > > > > > > > > > > > > > > getting
>> > > > > > > > > > > > > > > > > > > involved,
>> > > > > > > > > > > > > > > > > > > > > > > > despite
>> > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor
>> opening a
>> > > > > > dedicated
>> > > > > > > > > > > > discussion
>> > > > > > > > > > > > > > > thread
>> > > > > > > > > > > > > > > > > > > about the
>> > > > > > > > > > > > > > > > > > > > > > > > > > contribution
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and
>> > getting
>> > > > > > several
>> > > > > > > > +1's
>> > > > > > > > > > > from
>> > > > > > > > > > > > > > > > > committers.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really
>> make
>> > > > sure
>> > > > > > that
>> > > > > > > > if
>> > > > > > > > > we
>> > > > > > > > > > > > > > > > > welcome/approve
>> > > > > > > > > > > > > > > > > > > such
>> > > > > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it
>> will
>> > > > > actually
>> > > > > > > get
>> > > > > > > > > the
>> > > > > > > > > > > > > attention
>> > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > deserves.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm
>> inclined
>> > > to
>> > > > > > > > recommend
>> > > > > > > > > > > > > maintaining
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > > > > > > > outside
>> > > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could
>> link
>> > to
>> > > > it
>> > > > > > from
>> > > > > > > > the
>> > > > > > > > > > > > > > > documentation
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > give
>> > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the upcoming
>> > page
>> > > > for
>> > > > > > > > sharing
>> > > > > > > > > > > > > artifacts
>> > > > > > > > > > > > > > > > among
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > community
>> > > > > > > > > > > > > > > > > > > > > > > > > > (what's
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of that
>> > > > anyway?),
>> > > > > > > this
>> > > > > > > > > may
>> > > > > > > > > > > be a
>> > > > > > > > > > > > > > > better
>> > > > > > > > > > > > > > > > > > > option.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019
>> 10:16,
>> > > Till
>> > > > > > > > Rohrmann
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot for
>> > > > starting
>> > > > > > > this
>> > > > > > > > > > > > discussion
>> > > > > > > > > > > > > > > > Yijie.
>> > > > > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > > > > > think
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector would
>> be
>> > a
>> > > > very
>> > > > > > > > > valuable
>> > > > > > > > > > > > > addition
>> > > > > > > > > > > > > > > > since
>> > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > becomes
>> > > > > > > > > > > > > > > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular and
>> it
>> > > > would
>> > > > > > > > further
>> > > > > > > > > > > > expand
>> > > > > > > > > > > > > > > > Flink's
>> > > > > > > > > > > > > > > > > > > > > > > > > interoperability.
>> > > > > > > > > > > > > > > > > > > > > > > > > > Also
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project
>> > > > > perspective
>> > > > > > it
>> > > > > > > > > makes
>> > > > > > > > > > > > > sense for
>> > > > > > > > > > > > > > > > me
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > place
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream
>> > > project.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
>> > > > concern/question
>> > > > > is
>> > > > > > > how
>> > > > > > > > > can
>> > > > > > > > > > > the
>> > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > community
>> > > > > > > > > > > > > > > > > > > > > > > > maintain
>> > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We
>> have
>> > > seen
>> > > > > in
>> > > > > > > the
>> > > > > > > > > past
>> > > > > > > > > > > > that
>> > > > > > > > > > > > > > > > > connectors
>> > > > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > most
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively
>> developed
>> > > > > > components
>> > > > > > > > > > because
>> > > > > > > > > > > > > they
>> > > > > > > > > > > > > > > need
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > kept
>> > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > sync
>> > > > > > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external system
>> and
>> > > > with
>> > > > > > > Flink.
>> > > > > > > > > > Given
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > community
>> > > > > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
>> > > > maintaining,
>> > > > > > > > > improving
>> > > > > > > > > > > and
>> > > > > > > > > > > > > > > > evolving
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > connector,
>> > > > > > > > > > > > > > > > > > > > > > > > > I'm
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic that
>> we
>> > > can
>> > > > > > > achieve
>> > > > > > > > > > this.
>> > > > > > > > > > > > > Hence,
>> > > > > > > > > > > > > > > +1
>> > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > contributing
>> > > > > > > > > > > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > > > > > > > > > back
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
>> 2019
>> > > at
>> > > > > 2:03
>> > > > > > > AM
>> > > > > > > > > > Sijie
>> > > > > > > > > > > > Guo
>> > > > > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was the
>> > main
>> > > > > > driver
>> > > > > > > > > behind
>> > > > > > > > > > > > > > > FLINK-9641
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168,
>> > > > > > > > > > > > > > > > > > > > > > > > > let
>> > > > > > > > > > > > > > > > > > > > > > > > > > me
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more context
>> > on
>> > > > > this.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 and
>> > > > > FLINK-9168
>> > > > > > > was
>> > > > > > > > > > > created
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > bringing
>> > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > > > > > > source
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for Flink.
>> > The
>> > > > > > > > integration
>> > > > > > > > > > was
>> > > > > > > > > > > > done
>> > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > 1.6.0.
>> > > > > > > > > > > > > > > > > > > > > > > We
>> > > > > > > > > > > > > > > > > > > > > > > > > > sent out
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests about a
>> > > year
>> > > > > ago
>> > > > > > > and
>> > > > > > > > we
>> > > > > > > > > > > ended
>> > > > > > > > > > > > > up
>> > > > > > > > > > > > > > > > > > > maintaining
>> > > > > > > > > > > > > > > > > > > > > > those
>> > > > > > > > > > > > > > > > > > > > > > > > > > connectors
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for
>> Pulsar
>> > > > users
>> > > > > to
>> > > > > > > use
>> > > > > > > > > > Flink
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > process
>> > > > > > > > > > > > > > > > > > > event
>> > > > > > > > > > > > > > > > > > > > > > > streams
>> > > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
>> > > > > > > > > > > > > > > > > > > > > > > > > ).
>> > > > > > > > > > > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 integration
>> is
>> > > > > pretty
>> > > > > > > > simple
>> > > > > > > > > > and
>> > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > no
>> > > > > > > > > > > > > > > > > > > schema
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past
>> year,
>> > we
>> > > > > have
>> > > > > > > > made a
>> > > > > > > > > > lot
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > changes
>> > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > brought
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar schema as
>> > the
>> > > > > > > > first-class
>> > > > > > > > > > > > > citizen in
>> > > > > > > > > > > > > > > > > > Pulsar.
>> > > > > > > > > > > > > > > > > > > We
>> > > > > > > > > > > > > > > > > > > > > > also
>> > > > > > > > > > > > > > > > > > > > > > > > > > integrated
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other computing
>> > > > engines
>> > > > > > for
>> > > > > > > > > > > processing
>> > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > event
>> > > > > > > > > > > > > > > > > > > > > > streams
>> > > > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to
>> > rethink
>> > > > how
>> > > > > > to
>> > > > > > > > > > > integrate
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > best
>> > > > > > > > > > > > > > > > > > > > > > > > > way.
>> > > > > > > > > > > > > > > > > > > > > > > > > > Then
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement the
>> > > > > > pulsar-flink
>> > > > > > > > > > > > connectors
>> > > > > > > > > > > > > from
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > ground
>> > > > > > > > > > > > > > > > > > > > > up
>> > > > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > > > > schema
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table API
>> > and
>> > > > > > catalog
>> > > > > > > > API
>> > > > > > > > > as
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > first-class
>> > > > > > > > > > > > > > > > > > > > > citizen
>> > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration.
>> With
>> > > that
>> > > > > > being
>> > > > > > > > > said,
>> > > > > > > > > > > in
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > > > pulsar-flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can register
>> > > > pulsar
>> > > > > > as a
>> > > > > > > > > flink
>> > > > > > > > > > > > > catalog
>> > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > query /
>> > > > > > > > > > > > > > > > > > > > > > > process
>> > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > event
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using
>> > Flink
>> > > > SQL.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an
>> example
>> > > > about
>> > > > > > how
>> > > > > > > > to
>> > > > > > > > > > use
>> > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > catalog:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has also
>> > > > written a
>> > > > > > > blog
>> > > > > > > > > post
>> > > > > > > > > > > > > > > explaining
>> > > > > > > > > > > > > > > > > why
>> > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > > > > > re-implement
>> > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector with
>> > Flink
>> > > > 1.9
>> > > > > > and
>> > > > > > > > > what
>> > > > > > > > > > > are
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > changes
>> > > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > > made
>> > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe
>> Pulsar
>> > is
>> > > > not
>> > > > > > > just
>> > > > > > > > a
>> > > > > > > > > > > simple
>> > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > sink
>> > > > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > > source
>> > > > > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > It
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can be
>> a
>> > > > fully
>> > > > > > > > > integrated
>> > > > > > > > > > > > > streaming
>> > > > > > > > > > > > > > > > > data
>> > > > > > > > > > > > > > > > > > > > > storage
>> > > > > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink,
>> > source,
>> > > > > > > > > > schema/catalog
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > state).
>> > > > > > > > > > > > > > > > > > The
>> > > > > > > > > > > > > > > > > > > > > > > > combination
>> > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar can
>> > > create
>> > > > a
>> > > > > > > great
>> > > > > > > > > > > > streaming
>> > > > > > > > > > > > > > > > > warehouse
>> > > > > > > > > > > > > > > > > > > > > > > architecture
>> > > > > > > > > > > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streaming-first,
>> > > > unified
>> > > > > > > data
>> > > > > > > > > > > > > processing.
>> > > > > > > > > > > > > > > > Since
>> > > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > > > > > talking
>> > > > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute
>> Pulsar
>> > > > > > > integration
>> > > > > > > > to
>> > > > > > > > > > > Flink
>> > > > > > > > > > > > > here,
>> > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > also
>> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain,
>> improve
>> > > and
>> > > > > > evolve
>> > > > > > > > the
>> > > > > > > > > > > > > integration
>> > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > help
>> > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both
>> Flink
>> > > and
>> > > > > > > Pulsar.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this give
>> > you a
>> > > > bit
>> > > > > > > more
>> > > > > > > > > > > > background
>> > > > > > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. Let
>> > me
>> > > > know
>> > > > > > > what
>> > > > > > > > > are
>> > > > > > > > > > > your
>> > > > > > > > > > > > > > > > thoughts.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep 3,
>> > 2019
>> > > at
>> > > > > > 11:54
>> > > > > > > > AM
>> > > > > > > > > > Yun
>> > > > > > > > > > > > > Tang <
>> > > > > > > > > > > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see that
>> > > Pulsar
>> > > > > > > becomes
>> > > > > > > > > > more
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > more
>> > > > > > > > > > > > > > > > > > popular
>> > > > > > > > > > > > > > > > > > > > > > > recently
>> > > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > very
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more
>> > people
>> > > > > > willing
>> > > > > > > to
>> > > > > > > > > > > > > contribute to
>> > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > ecosystem.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any
>> > further
>> > > > > > > > discussion,
>> > > > > > > > > > > would
>> > > > > > > > > > > > > you
>> > > > > > > > > > > > > > > > please
>> > > > > > > > > > > > > > > > > > > give
>> > > > > > > > > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > > > > > > > > > > > explanation
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the
>> relationship
>> > > > > between
>> > > > > > > this
>> > > > > > > > > > > thread
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > current
>> > > > > > > > > > > > > > > > > > > > > existing
>> > > > > > > > > > > > > > > > > > > > > > > > JIRAs
>> > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] and
>> > sink
>> > > > [2]
>> > > > > > > > > > connector?
>> > > > > > > > > > > > > Will the
>> > > > > > > > > > > > > > > > > > > > > contribution
>> > > > > > > > > > > > > > > > > > > > > > > > > contains
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > part
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or
>> > > totally
>> > > > > > > > different
>> > > > > > > > > > > > > > > > implementation?
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
>> > > > > > > > > > > > > > > > > > >
>> > > > > https://issues.apache.org/jira/browse/FLINK-9641
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
>> > > > > > > > > > > > > > > > > > >
>> > > > > https://issues.apache.org/jira/browse/FLINK-9168
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > ________________________________
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie
>> Shen
>> > <
>> > > > > > > > > > > > > > > [hidden email]>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: Tuesday,
>> > > > > September
>> > > > > > 3,
>> > > > > > > > > 2019
>> > > > > > > > > > > > 13:57
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
>> > > > > [hidden email]
>> > > > > > <
>> > > > > > > > > > > > > > > > [hidden email]
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject:
>> > [DISCUSS]
>> > > > > > > Contribute
>> > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > > > back
>> > > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink
>> > > Community!
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would like to
>> > > open
>> > > > > the
>> > > > > > > > > > discussion
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > contributing
>> > > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > Flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector [0]
>> > back
>> > > to
>> > > > > > > Flink.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
>> > > > introduction
>> > > > > > to
>> > > > > > > > > Apache
>> > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache
>> Pulsar[1]
>> > > is a
>> > > > > > > > > > multi-tenant,
>> > > > > > > > > > > > > > > > > > > high-performance
>> > > > > > > > > > > > > > > > > > > > > > > > > distributed
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub
>> messaging
>> > > > > system.
>> > > > > > > > > Pulsar
>> > > > > > > > > > > > > includes
>> > > > > > > > > > > > > > > > > > multiple
>> > > > > > > > > > > > > > > > > > > > > > features
>> > > > > > > > > > > > > > > > > > > > > > > > > such
>> > > > > > > > > > > > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native support
>> > for
>> > > > > > multiple
>> > > > > > > > > > > clusters
>> > > > > > > > > > > > > in a
>> > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > instance,
>> > > > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
>> > > > > geo-replication
>> > > > > > of
>> > > > > > > > > > > messages
>> > > > > > > > > > > > > across
>> > > > > > > > > > > > > > > > > > > clusters,
>> > > > > > > > > > > > > > > > > > > > > > very
>> > > > > > > > > > > > > > > > > > > > > > > > low
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and end-to-end
>> > > > latency,
>> > > > > > > > > seamless
>> > > > > > > > > > > > > > > scalability
>> > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > over a
>> > > > > > > > > > > > > > > > > > > > > > > > million
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and guaranteed
>> > > > message
>> > > > > > > > delivery
>> > > > > > > > > > > with
>> > > > > > > > > > > > > > > > persistent
>> > > > > > > > > > > > > > > > > > > message
>> > > > > > > > > > > > > > > > > > > > > > > > storage
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by
>> > Apache
>> > > > > > > > BookKeeper.
>> > > > > > > > > > > > > Nowadays,
>> > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > has
>> > > > > > > > > > > > > > > > > > > > > been
>> > > > > > > > > > > > > > > > > > > > > > > > > adopted
>> > > > > > > > > > > > > > > > > > > > > > > > > > by
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and more
>> > > > > > companies[2].
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The status
>> of
>> > > > Pulsar
>> > > > > > > Flink
>> > > > > > > > > > > > connector
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar
>> Flink
>> > > > > > connector
>> > > > > > > we
>> > > > > > > > > are
>> > > > > > > > > > > > > planning
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > contribute
>> > > > > > > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > > > > > > > built
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 and
>> > > > Pulsar
>> > > > > > > 2.4.0.
>> > > > > > > > > The
>> > > > > > > > > > > > main
>> > > > > > > > > > > > > > > > features
>> > > > > > > > > > > > > > > > > > > are:
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as a
>> > > > streaming
>> > > > > > > > source
>> > > > > > > > > > with
>> > > > > > > > > > > > > > > > > exactly-once
>> > > > > > > > > > > > > > > > > > > > > > guarantee.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink
>> streaming
>> > > > > results
>> > > > > > to
>> > > > > > > > > > Pulsar
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > at-least-once
>> > > > > > > > > > > > > > > > > > > > > > > > > semantics.
>> > > > > > > > > > > > > > > > > > > > > > > > > > (We
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update
>> this
>> > > to
>> > > > > > > > > exactly-once
>> > > > > > > > > > > as
>> > > > > > > > > > > > > well
>> > > > > > > > > > > > > > > > when
>> > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > gets
>> > > > > > > > > > > > > > > > > > > > > > > > all
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction
>> > > features
>> > > > > > ready
>> > > > > > > in
>> > > > > > > > > its
>> > > > > > > > > > > > 2.5.0
>> > > > > > > > > > > > > > > > > version)
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon
>> > Flink
>> > > > new
>> > > > > > > Table
>> > > > > > > > > API
>> > > > > > > > > > > Type
>> > > > > > > > > > > > > > > system
>> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
>> > > > > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> automatically
>> > > > > > (de)serialize
>> > > > > > > > > > > messages
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > help
>> > > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > Pulsar
>> > > > > > > > > > > > > > > > > > > > > > > > > > schema.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate
>> with
>> > > > Flink
>> > > > > > new
>> > > > > > > > > > Catalog
>> > > > > > > > > > > > API
>> > > > > > > > > > > > > > > > > > > (FLIP-30[4]),
>> > > > > > > > > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > > > > > > > > > enables
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of Pulsar
>> > > topics
>> > > > as
>> > > > > > > > tables
>> > > > > > > > > in
>> > > > > > > > > > > > > Table API
>> > > > > > > > > > > > > > > > as
>> > > > > > > > > > > > > > > > > > > well as
>> > > > > > > > > > > > > > > > > > > > > > SQL
>> > > > > > > > > > > > > > > > > > > > > > > > > > client.
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
>> > > > > > > > > > > > > > > > >
>> https://github.com/streamnative/pulsar-flink
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
>> > > > > > > > https://pulsar.apache.org/
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
>> > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>