[DISCUSS] Contribute Pulsar Flink connector back to Flink

[Till Rohrmann]

What is the conclusion of this discussion? Are we gonna merge the old
source/sink or do we believe that FLIP-27 will be completed in time so that
we can merge the new source right away?

Cheers,
Till

On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <[hidden email]> wrote:

>
>
> unscribe
>
>
>
>
>
>
> At 2019-09-24 08:53:25, "Becket Qin" <[hidden email]> wrote:
> >Thanks, Stephan.
> >
> >Sounds good to me. We can still try our best to get new Pulsar connector
> in
> >Flink 1.10. In case we do not have time to do that, we will prominently
> >link the Pulsar connector from the Flink connector docs.
> >
> >Thanks,
> >
> >Jiangjie (Becket) Qin
> >
> >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <[hidden email]> wrote:
> >
> >> Okay, I see your point, Becket.
> >>
> >> Then let us prominently link the Pulsar connector from the Flink
> connector
> >> docs then, so that users can find it easily.
> >>
> >> As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute
> a
> >> new connector.
> >>
> >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]>
> wrote:
> >>
> >> > Hi Stephan,
> >> >
> >> > I have no doubt about the value of adding Pulsar connector to Flink
> repo.
> >> > My concern is about how exactly we are going to do it.
> >> >
> >> > As mentioned before, I believe that we can handle connectors more
> >> > > pragmatically and less strict than the core of Flink, if it helps
> >> > unlocking
> >> > > users.
> >> >
> >> > I can see the benefit of being less restrict for the initial connector
> >> code
> >> > adoption. However, I don't think we should be less restrict on the
> >> > maintenance commitment once the code is in Flink repo. It only makes
> >> sense
> >> > to check in something and ask users to use if we plan to maintain it.
> >> >
> >> > If I understand correctly, the current plan so far is following:
> >> > 1. release 1.10
> >> >    - Check in Pulsar connector on old interface and label it as beta
> >> > version.
> >> >    - encourage users to try it and report bugs.
> >> > 2. release 1.11
> >> >    - Check in Pulsar connector on new interface (a.k.a new Pulsar
> >> > connector) and label it as beta version
> >> >    - Deprecate the old Pulsar connector
> >> >    - Fix bugs reported on old Pulsar connector from release 1.10
> >> >    - Ask users to migrate from old Pulsar connector to new Pulsar
> >> connector
> >> > 3. release 1.12
> >> >    - Announce end of support for old Pulsar connector and remove the
> code
> >> >    - Fix bugs reported on new Pulsar connector.
> >> >
> >> > If this is the plan, it seems neither Flink nor the users trying the
> old
> >> > Pulsar connector will benefit from this experimental old Pulsar
> >> connector,
> >> > because whatever feedbacks we got or bugs we fix on the old Pulsar
> >> > connector are immediately thrown away in one or two releases.
> >> >
> >> > If we check in the old Pulsar connector right now, the only option I
> see
> >> is
> >> > to maintain it for a while (e.g. a year or more). IMO, the immediate
> >> > deprecation and code removal hurts the users much more than asking
> them
> >> to
> >> > wait for another release. I personally think that we can avoid this
> >> > maintenance burden by going directly to the new Pulsar connector,
> >> > especially given that users can still use the connector even if they
> are
> >> > not in Flink repo. That said, I am OK with maintaining both old and
> new
> >> > Pulsar connector if we believe that having the Pulsar connector
> available
> >> > right now in Flink repo is more important.
> >> >
> >> > Thanks,
> >> >
> >> > Jiangjie (Becket) Qin
> >> >
> >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]>
> wrote:
> >> >
> >> > > My assumption is as Sijie's, that once the connector is either part
> of
> >> > > Flink, or part of the streamnative repo. No double maintenance.
> >> > >
> >> > > I feel this discussion is very much caught in problems that are all
> >> > > solvable if we want to solve them.
> >> > > Maybe we can think what our goal for users and the communities is?
> >> > >
> >> > >   - Do we want to help build a relationship between the Pulsar and
> >> Flink
> >> > > open source communities?
> >> > >   - Will users find a connector in the streamnative repository?
> >> > >   - Will users trust a connector that is not part of Flink as much?
> >> > >
> >> > > And then decide what is best according to the overall goals there.
> >> > > As mentioned before, I believe that we can handle connectors more
> >> > > pragmatically and less strict than the core of Flink, if it helps
> >> > unlocking
> >> > > users.
> >> > >
> >> > > Best,
> >> > > Stephan
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]>
> wrote:
> >> > >
> >> > > > Thanks Becket.
> >> > > >
> >> > > > I think it is better for the Flink community to judge the
> benefits of
> >> > > doing
> >> > > > this. I was trying to provide some views from outsiders.
> >> > > >
> >> > > > Thanks,
> >> > > > Sijie
> >> > > >
> >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <[hidden email]
> >
> >> > > wrote:
> >> > > >
> >> > > > > Hi Sijie,
> >> > > > >
> >> > > > > Yes, we will have to support existing old connectors and new
> >> > connectors
> >> > > > in
> >> > > > > parallel for a while. We have to take that maintenance overhead
> >> > because
> >> > > > > existing connectors have been used by the users for a long
> time. I
> >> > > guess
> >> > > > It
> >> > > > > may take at least a year for us to fully remove the old
> connectors.
> >> > > > >
> >> > > > > Process wise, we can do the same for Pulsar connector. But I am
> not
> >> > > sure
> >> > > > if
> >> > > > > we want to have the same burden on Pulsar connector, and I would
> >> like
> >> > > to
> >> > > > > understand the benefit of doing that.
> >> > > > >
> >> > > > > For users, the benefit of having the old Pulsar connector
> checked
> >> in
> >> > > > seems
> >> > > > > limited because 1) that code base will be immediately
> deprecated in
> >> > the
> >> > > > > next release in 3-4 months; 2) users can always use it even if
> it
> >> is
> >> > > not
> >> > > > in
> >> > > > > the Flink code base. Admittedly it is not as convenient as
> having
> >> it
> >> > in
> >> > > > > Flink code base, but doesn't seem super either. And after 3-4
> >> months,
> >> > > > users
> >> > > > > can just use the new connector in Flink repo.
> >> > > > >
> >> > > > > For Flink developers, the old connector code base is not
> something
> >> > that
> >> > > > we
> >> > > > > want to evolve later. Instead, these code will be deprecated and
> >> > > > > removed. So why do we want to get a beta version out to attract
> >> > people
> >> > > to
> >> > > > > use something we don't want to maintain?
> >> > > > >
> >> > > > > Thanks,
> >> > > > >
> >> > > > > Jiangjie (Becket) Qin
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <[hidden email]>
> >> > wrote:
> >> > > > >
> >> > > > > > Thanks everyone here. Sorry for jumping into the discussion
> here.
> >> > > > > >
> >> > > > > > I am not very familiar about the deprecation process in Flink.
> >> If I
> >> > > > > > misunderstood the process, please fix me.
> >> > > > > >
> >> > > > > > As far as I understand, FLIP-27 is introducing a new unified
> API
> >> > for
> >> > > > > > connectors. After it introduces the new API
> >> > > > > > and before moving all the existing connectors from old API to
> new
> >> > > API,
> >> > > > > both
> >> > > > > > old ApI and new API will co-exist
> >> > > > > > for a while until Flink moves all existing connectors to new
> API.
> >> > So
> >> > > > the
> >> > > > > > Pulsar connector (using old API) can
> >> > > > > > follow the deprecation process with other connector using old
> API
> >> > and
> >> > > > the
> >> > > > > > deprecation of old API, no?
> >> > > > > >
> >> > > > > > If that's the case, I think contributing the current connector
> >> back
> >> > > to
> >> > > > > > Flink rather than maintaining it outside Flink
> >> > > > > > would provide a bit more benefits. We can deprecate the
> existing
> >> > > > > > streamnative/pulsar-flink repo and point the users
> >> > > > > > to use the connector in Flink repo. So all the review
> processes
> >> > will
> >> > > > > happen
> >> > > > > > within Flink for both old connector and
> >> > > > > > new connector. It also reduces the confusions for the users as
> >> the
> >> > > > > > documentation and code base happen in one place.
> >> > > > > >
> >> > > > > > Thoughts?
> >> > > > > >
> >> > > > > > - Sijie
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <
> >> [hidden email]>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks for the explanation, Stephan. I have a few questions
> /
> >> > > > thoughts.
> >> > > > > > >
> >> > > > > > > So that means we will remove the old connector without a
> major
> >> > > > version
> >> > > > > > > bump, is that correct?
> >> > > > > > >
> >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11
> >> connectors
> >> > > > will
> >> > > > > > > always work because we saw some dependency class collisions
> in
> >> > the
> >> > > > > past.
> >> > > > > > To
> >> > > > > > > make it safe we may have to maintain the old code for one
> more
> >> > > > release.
> >> > > > > > >
> >> > > > > > > To be honest I am still wondering if we have to put the old
> >> > > connector
> >> > > > > in
> >> > > > > > > Flink repo. if we check in the old connector to Flink. We
> will
> >> > end
> >> > > up
> >> > > > > in
> >> > > > > > > the following situation:
> >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> >> > > > > > > 2. Old connector in Flink Repo, which may be different from
> the
> >> > one
> >> > > > in
> >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in
> >> 1.12)
> >> > > > > > > 3. New connector in Flink Repo.
> >> > > > > > >
> >> > > > > > > We need to think about how to make the users in each case
> >> happy.
> >> > > > > > > - For users of (1), I assume Sijie and Yijie will have to
> >> > maintain
> >> > > > the
> >> > > > > > code
> >> > > > > > > a bit longer for its own compatibility even after we have
> (2).
> >> In
> >> > > > that
> >> > > > > > > case, bugs found in old connector may or may not need to be
> >> fixed
> >> > > in
> >> > > > > both
> >> > > > > > > Flink and the streamnative/pulsar-flink repo.
> >> > > > > > > - For users of (2), will we provide bug fixes? If we do, it
> >> will
> >> > > be a
> >> > > > > > > little awkward because those bug fixes will be immediately
> >> > > deprecated
> >> > > > > in
> >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking
> users
> >> to
> >> > > > > migrate
> >> > > > > > > away from the bug fix. After Flink 1.12, users may still
> have
> >> to
> >> > > > switch
> >> > > > > > to
> >> > > > > > > use (3) due to the potential dependency class conflicts
> >> mentioned
> >> > > > > above.
> >> > > > > > > - Users of (3) have a much easier life and don't need to
> worry
> >> > too
> >> > > > > much.
> >> > > > > > >
> >> > > > > > > The above story seems a little complicated to tell. I think
> it
> >> > will
> >> > > > be
> >> > > > > > much
> >> > > > > > > easier to not have (2) at all.
> >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> >> > > > > > > 3. New connector in Flink Repo.
> >> > > > > > >
> >> > > > > > > - Old connector will only be maintained in
> >> > > streamnative/pulsar-flink
> >> > > > > repo
> >> > > > > > > until it is fully deprecated. Users can always use the
> existing
> >> > > > Pulsar
> >> > > > > > > connector in that repo.
> >> > > > > > > - New connector will be in Flink repo and maintained like
> the
> >> > other
> >> > > > > > > connectors.
> >> > > > > > >
> >> > > > > > > This seems much simpler for users understand and they will
> not
> >> be
> >> > > > > blocked
> >> > > > > > > from using the old connector. If the concern is about the
> >> quality
> >> > > of
> >> > > > > the
> >> > > > > > > connector in streamnative/pulsar-flink repo, is it enough
> for
> >> us
> >> > > just
> >> > > > > to
> >> > > > > > > review the code in streamnative/pulsar-flink connector to
> make
> >> > sure
> >> > > > it
> >> > > > > > > looks good from Flink's perspective?
> >> > > > > > >
> >> > > > > > > What do you think?
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > >
> >> > > > > > > Jiangjie (Becket) Qin
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <
> [hidden email]
> >> >
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > > > My take would be the following:
> >> > > > > > > >
> >> > > > > > > >   - If we merge the connector now and replace it with a
> >> FLIP-27
> >> > > > > version
> >> > > > > > > > before the 1.10 release, then we need no deprecation
> process
> >> > > > > > > >   - If we don't manage to replace it with a FLIP-27
> version
> >> > > before
> >> > > > > the
> >> > > > > > > 1.10
> >> > > > > > > > release, than it is good that we have the other version,
> so
> >> no
> >> > > > users
> >> > > > > > get
> >> > > > > > > > blocked.
> >> > > > > > > >
> >> > > > > > > > In the latter case we can see how we want to do it.
> Immediate
> >> > > > removal
> >> > > > > > of
> >> > > > > > > > the old version or deprecation label and keeping it for
> one
> >> > more
> >> > > > > > release.
> >> > > > > > > > Given that you should be able to use a Flink 1.10
> connector
> >> > with
> >> > > > > Flink
> >> > > > > > > 1.11
> >> > > > > > > > as well (stable public APIs) there is also a workaround if
> >> you
> >> > > need
> >> > > > > an
> >> > > > > > > old
> >> > > > > > > > connector in a newer version. So immediate removal might
> even
> >> > be
> >> > > > > > > feasible.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
> >> > > [hidden email]>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Stephan,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the clarification. I completely agree with
> you
> >> and
> >> > > > > Thomas
> >> > > > > > on
> >> > > > > > > > the
> >> > > > > > > > > process of adding connectors to Flink repo. However, I
> am
> >> > > > wondering
> >> > > > > > > what
> >> > > > > > > > is
> >> > > > > > > > > the deprecation process? Given the main concern here was
> >> that
> >> > > we
> >> > > > > may
> >> > > > > > > have
> >> > > > > > > > > to maintain two Pulsar connector code bases until the
> old
> >> one
> >> > > is
> >> > > > > > > removed
> >> > > > > > > > > from the repo, it would be good to know how long we
> have to
> >> > do
> >> > > > > that.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
> >> > [hidden email]
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Some quick thoughts on the connector contribution
> >> process.
> >> > I
> >> > > > > > > basically
> >> > > > > > > > > > reiterate here what Thomas mentioned in another thread
> >> > about
> >> > > > the
> >> > > > > > > > Kinesis
> >> > > > > > > > > > connector.
> >> > > > > > > > > >
> >> > > > > > > > > > For connectors, we should favor a low-overhead
> >> contribution
> >> > > > > > process,
> >> > > > > > > > and
> >> > > > > > > > > > accept user code and changes more readily than in the
> >> core
> >> > > > > system.
> >> > > > > > > > > > That is because connectors have both a big variety of
> >> > > scenarios
> >> > > > > > they
> >> > > > > > > > get
> >> > > > > > > > > > used in (only through use and many small
> contributions do
> >> > > they
> >> > > > > > become
> >> > > > > > > > > > really useful over time) and at the same time, and
> >> > committers
> >> > > > do
> >> > > > > > not
> >> > > > > > > > use
> >> > > > > > > > > > the connector themselves and usually cannot foresee
> too
> >> > well
> >> > > > what
> >> > > > > > is
> >> > > > > > > > > > needed.
> >> > > > > > > > > >
> >> > > > > > > > > > Further more, a missing connector (or connector
> feature)
> >> is
> >> > > > > often a
> >> > > > > > > > > bigger
> >> > > > > > > > > > show stopper for users than a missing API or system
> >> > feature.
> >> > > > > > > > > >
> >> > > > > > > > > > Along these lines of thougt, the conclusion would be
> to
> >> > take
> >> > > > the
> >> > > > > > > Pulsar
> >> > > > > > > > > > connector now, focus the review on
> >> legal/dependencies/rough
> >> > > > code
> >> > > > > > > style
> >> > > > > > > > > and
> >> > > > > > > > > > conventions, label it as "beta" (in the sense of "new
> >> code"
> >> > > > that
> >> > > > > is
> >> > > > > > > > "not
> >> > > > > > > > > > yet tested through longer use") and go ahead. And then
> >> > evolve
> >> > > > it
> >> > > > > > > > quickly
> >> > > > > > > > > > without putting formal blockers in the way, meaning
> also
> >> > > > adding a
> >> > > > > > new
> >> > > > > > > > > FLIP
> >> > > > > > > > > > 27 version when it is there.
> >> > > > > > > > > >
> >> > > > > > > > > > Best,
> >> > > > > > > > > > Stephan
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> >> > > > [hidden email]
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Yijie,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Could you please follow the FLIP process to start a
> new
> >> > > FLIP
> >> > > > > > > > > [DISCUSSION]
> >> > > > > > > > > > > thread in the mailing list?
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> >> > > > > > > > > > >
> >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list
> now.
> >> So
> >> > > > there
> >> > > > > > is a
> >> > > > > > > > > FLIP
> >> > > > > > > > > > > number collision. Can you change the FLIP number to
> 72?
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> >> > > > > [hidden email]>
> >> > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi Yijie,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP.
> >> > > > > > > > > > > > Would you mind enabling comments/suggestions on
> the
> >> > > google
> >> > > > > doc
> >> > > > > > > > link?
> >> > > > > > > > > > This
> >> > > > > > > > > > > > way the contributors from the community can
> comment
> >> on
> >> > > the
> >> > > > > doc.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Best,
> >> > > > > > > > > > > > Rong
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> >> > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Hello everyone,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > I've drafted a FLIP that describes the current
> >> design
> >> > > of
> >> > > > > the
> >> > > > > > > > Pulsar
> >> > > > > > > > > > > > > connector:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Please take a look and let me know what you
> think.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > Yijie
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> >> > > > > > > [hidden email]>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi All,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Sorry for joining the discussion late and
> thanks
> >> > > Yijie
> >> > > > &
> >> > > > > > > Sijie
> >> > > > > > > > > for
> >> > > > > > > > > > > > > driving
> >> > > > > > > > > > > > > > the discussion.
> >> > > > > > > > > > > > > > I also think the Pulsar connector would be a
> very
> >> > > > > valuable
> >> > > > > > > > > addition
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > > > Flink. I can also help out a bit on the review
> >> side
> >> > > :-)
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Regarding the timeline, I also share concerns
> >> with
> >> > > > Becket
> >> > > > > > on
> >> > > > > > > > the
> >> > > > > > > > > > > > > > relationship between the new Pulsar connector
> and
> >> > > > > FLIP-27.
> >> > > > > > > > > > > > > > There's also another discussion just started
> by
> >> > > Stephan
> >> > > > > on
> >> > > > > > > > > dropping
> >> > > > > > > > > > > > Kafka
> >> > > > > > > > > > > > > > 9/10 support on next Flink release [1].
> Although
> >> > the
> >> > > > > > > situation
> >> > > > > > > > > is
> >> > > > > > > > > > > > > somewhat
> >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has been
> in
> >> > Flink
> >> > > > for
> >> > > > > > > > almost
> >> > > > > > > > > > 3-4
> >> > > > > > > > > > > > > years,
> >> > > > > > > > > > > > > > based on the discussion I am not sure if a
> major
> >> > > > version
> >> > > > > > > > release
> >> > > > > > > > > > is a
> >> > > > > > > > > > > > > > requirement for removing old connector
> supports.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I think there shouldn't be a blocker if we
> agree
> >> > the
> >> > > > old
> >> > > > > > > > > connector
> >> > > > > > > > > > > will
> >> > > > > > > > > > > > > be
> >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is
> >> > there.
> >> > > > As
> >> > > > > > > > Stephan
> >> > > > > > > > > > > > stated,
> >> > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > is easier to contribute the source sooner and
> >> > adjust
> >> > > it
> >> > > > > > > later.
> >> > > > > > > > > > > > > > We should also ensure we clearly communicate
> the
> >> > > > message:
> >> > > > > > for
> >> > > > > > > > > > > example,
> >> > > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27
> >> > > > connector
> >> > > > > > page
> >> > > > > > > > of
> >> > > > > > > > > > the
> >> > > > > > > > > > > > > > website, documentations, etc. Any other
> thoughts?
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > --
> >> > > > > > > > > > > > > > Rong
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > [1]
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> >> > > > > > > > [hidden email]
> >> > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Technically speaking, removing the old
> >> connector
> >> > > code
> >> > > > > is
> >> > > > > > a
> >> > > > > > > > > > > backwards
> >> > > > > > > > > > > > > > > incompatible change which requires a major
> >> > version
> >> > > > > bump,
> >> > > > > > > i.e.
> >> > > > > > > > > > Flink
> >> > > > > > > > > > > > > 2.x.
> >> > > > > > > > > > > > > > > Given that we don't have a clear plan on
> when
> >> to
> >> > > have
> >> > > > > the
> >> > > > > > > > next
> >> > > > > > > > > > > major
> >> > > > > > > > > > > > > > > version release, it seems unclear how long
> the
> >> > old
> >> > > > > > > connector
> >> > > > > > > > > code
> >> > > > > > > > > > > > will
> >> > > > > > > > > > > > > be
> >> > > > > > > > > > > > > > > there if we check it in right now. Or will
> we
> >> > > remove
> >> > > > > the
> >> > > > > > > old
> >> > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > without a major version bump? In any case,
> it
> >> > > sounds
> >> > > > > not
> >> > > > > > > > quite
> >> > > > > > > > > > user
> >> > > > > > > > > > > > > > > friendly to the those who might use the old
> >> > Pulsar
> >> > > > > > > > connector. I
> >> > > > > > > > > > am
> >> > > > > > > > > > > > not
> >> > > > > > > > > > > > > sure
> >> > > > > > > > > > > > > > > if it is worth these potential problems in
> >> order
> >> > to
> >> > > > > have
> >> > > > > > > the
> >> > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > source
> >> > > > > > > > > > > > > > > connector checked in one or two months
> earlier.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan
> Ewen <
> >> > > > > > > > [hidden email]
> >> > > > > > > > > >
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > Agreed, if we check in the old code, we
> >> should
> >> > > make
> >> > > > > it
> >> > > > > > > > clear
> >> > > > > > > > > > that
> >> > > > > > > > > > > > it
> >> > > > > > > > > > > > > will
> >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based
> >> version
> >> > > of
> >> > > > > the
> >> > > > > > > > > > connector
> >> > > > > > > > > > > is
> >> > > > > > > > > > > > > > > there.
> >> > > > > > > > > > > > > > > > We should not commit to maintaining the
> old
> >> > > > version,
> >> > > > > > that
> >> > > > > > > > > would
> >> > > > > > > > > > > be
> >> > > > > > > > > > > > > indeed
> >> > > > > > > > > > > > > > > > too much overhead.
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket
> Qin <
> >> > > > > > > > > > [hidden email]
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Hi Stephan,
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Thanks for the volunteering to help.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Yes, the overhead would just be review
> >> > > capacity.
> >> > > > In
> >> > > > > > > > fact, I
> >> > > > > > > > > > am
> >> > > > > > > > > > > > not
> >> > > > > > > > > > > > > > > > worrying
> >> > > > > > > > > > > > > > > > > too much about the review capacity.
> That is
> >> > > just
> >> > > > a
> >> > > > > > one
> >> > > > > > > > time
> >> > > > > > > > > > > cost.
> >> > > > > > > > > > > > > My
> >> > > > > > > > > > > > > > > > > concern is mainly about the long term
> >> burden.
> >> > > > > Assume
> >> > > > > > we
> >> > > > > > > > > have
> >> > > > > > > > > > > new
> >> > > > > > > > > > > > > source
> >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly added
> >> > Pulsar
> >> > > > > > > > connectors
> >> > > > > > > > > in
> >> > > > > > > > > > > old
> >> > > > > > > > > > > > > > > > > interface. Later on if we migrate
> Pulsar to
> >> > new
> >> > > > > > source
> >> > > > > > > > > > > interface,
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > old
> >> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated
> almost
> >> > > > > > immediately
> >> > > > > > > > > after
> >> > > > > > > > > > > > > checked
> >> > > > > > > > > > > > > > > in,
> >> > > > > > > > > > > > > > > > > but we may still have to maintain two
> code
> >> > > bases.
> >> > > > > For
> >> > > > > > > the
> >> > > > > > > > > > > > existing
> >> > > > > > > > > > > > > > > > > connectors, we have to do that anyways.
> But
> >> > it
> >> > > > > would
> >> > > > > > be
> >> > > > > > > > > good
> >> > > > > > > > > > to
> >> > > > > > > > > > > > > avoid
> >> > > > > > > > > > > > > > > > > introducing a new connector with the
> same
> >> > > > problem.
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan
> >> Ewen
> >> > <
> >> > > > > > > > > > [hidden email]
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Hi all!
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Nice to see this lively discussion
> about
> >> > the
> >> > > > > Pulsar
> >> > > > > > > > > > > connector.
> >> > > > > > > > > > > > > > > > > > Some thoughts on the open questions:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as
> a
> >> > > > community
> >> > > > > > > > package
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Looks like the discussion is more
> going
> >> > > towards
> >> > > > > > > > > > > contribution. I
> >> > > > > > > > > > > > > think
> >> > > > > > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > > > > is good, especially if we think that
> we
> >> > want
> >> > > to
> >> > > > > > > build a
> >> > > > > > > > > > > > similarly
> >> > > > > > > > > > > > > > > deep
> >> > > > > > > > > > > > > > > > > > integration with Pulsar as we have for
> >> > > example
> >> > > > > with
> >> > > > > > > > > Kafka.
> >> > > > > > > > > > > The
> >> > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > already looks like a more thorough
> >> > connector
> >> > > > than
> >> > > > > > > many
> >> > > > > > > > > > others
> >> > > > > > > > > > > > we
> >> > > > > > > > > > > > > have
> >> > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > the repository.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > With either a repo split, or the new
> >> build
> >> > > > > system,
> >> > > > > > I
> >> > > > > > > > hope
> >> > > > > > > > > > > that
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > build
> >> > > > > > > > > > > > > > > > > > overhead is not a problem.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > ## Committer Support
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Becket offered some help already, I
> can
> >> > also
> >> > > > > help a
> >> > > > > > > > bit.
> >> > > > > > > > > I
> >> > > > > > > > > > > hope
> >> > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > > > > between us, we can cover this.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually
> >> making
> >> > > some
> >> > > > > > > PoC-ing
> >> > > > > > > > > > > > > progress, but
> >> > > > > > > > > > > > > > > > > will
> >> > > > > > > > > > > > > > > > > > take 2 more months, I would estimate,
> >> > before
> >> > > it
> >> > > > > is
> >> > > > > > > > fully
> >> > > > > > > > > > > > > available.
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > If we want to be on the safe side with
> >> the
> >> > > > > > > > contribution,
> >> > > > > > > > > we
> >> > > > > > > > > > > > > should
> >> > > > > > > > > > > > > > > > > > contribute the source sooner and
> adjust
> >> it
> >> > > > later.
> >> > > > > > > That
> >> > > > > > > > > > would
> >> > > > > > > > > > > > also
> >> > > > > > > > > > > > > > > help
> >> > > > > > > > > > > > > > > > us
> >> > > > > > > > > > > > > > > > > > in case things get crazy towards the
> 1.10
> >> > > > feature
> >> > > > > > > > freeze
> >> > > > > > > > > > and
> >> > > > > > > > > > > it
> >> > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > > > be
> >> > > > > > > > > > > > > > > > > > hard to find time to review the new
> >> > changes.
> >> > > > > > > > > > > > > > > > > > What would be the overhead of
> >> contributing
> >> > > now?
> >> > > > > > Given
> >> > > > > > > > > that
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > code
> >> > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > already there, it looks like it would
> be
> >> > only
> >> > > > > > review
> >> > > > > > > > > > > capacity,
> >> > > > > > > > > > > > > right?
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > > > > > Stephan
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie
> >> > Shen <
> >> > > > > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > Hi everyone!
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > Thanks for your attention and the
> >> > promotion
> >> > > > of
> >> > > > > > this
> >> > > > > > > > > work.
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as
> >> > possible
> >> > > > for
> >> > > > > > more
> >> > > > > > > > > > > specific
> >> > > > > > > > > > > > > > > > > discussions.
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have
> not
> >> > > > reached
> >> > > > > a
> >> > > > > > > > > > consensus.
> >> > > > > > > > > > > > > > > > Therefore,
> >> > > > > > > > > > > > > > > > > > > I will explain all the
> functionalities
> >> of
> >> > > the
> >> > > > > > > > existing
> >> > > > > > > > > > > > > connector in
> >> > > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink,
> and
> >> > > > Catalog)
> >> > > > > to
> >> > > > > > > > > > continue
> >> > > > > > > > > > > > our
> >> > > > > > > > > > > > > > > > > > > discussions in FLIP.
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > Thanks for your kind help.
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > > > > > > Yijie
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM
> Becket
> >> > Qin
> >> > > <
> >> > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > Hi Sijie,
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is to
> have
> >> > > Pulsar
> >> > > > > > > > connector
> >> > > > > > > > > > in
> >> > > > > > > > > > > > > 1.10,
> >> > > > > > > > > > > > > > > how
> >> > > > > > > > > > > > > > > > > > about
> >> > > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > do the following:
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar
> >> connector
> >> > > to
> >> > > > > > Flink
> >> > > > > > > > main
> >> > > > > > > > > > > repo
> >> > > > > > > > > > > > > as it
> >> > > > > > > > > > > > > > > > is a
> >> > > > > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > > public interface to Flink main
> repo.
> >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink
> >> > right
> >> > > > away
> >> > > > > > as
> >> > > > > > > > > there
> >> > > > > > > > > > is
> >> > > > > > > > > > > > no
> >> > > > > > > > > > > > > > > change
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > sink interface so far.
> >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27.
> >> Flink
> >> > > 1.10
> >> > > > > is
> >> > > > > > > > going
> >> > > > > > > > > to
> >> > > > > > > > > > > be
> >> > > > > > > > > > > > > code
> >> > > > > > > > > > > > > > > > > freeze
> >> > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we give a
> >> month
> >> > to
> >> > > > the
> >> > > > > > > > > > development
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > review
> >> > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have
> >> > FLIP-27
> >> > > > by
> >> > > > > > late
> >> > > > > > > > > Oct.
> >> > > > > > > > > > > > There
> >> > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > still
> >> > > > > > > > > > > > > > > > > > 7
> >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it is
> >> doable.
> >> > > If
> >> > > > > > > FLIP-27
> >> > > > > > > > is
> >> > > > > > > > > > not
> >> > > > > > > > > > > > > ready
> >> > > > > > > > > > > > > > > by
> >> > > > > > > > > > > > > > > > > late
> >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check in
> >> Pulsar
> >> > > > > > connector
> >> > > > > > > > with
> >> > > > > > > > > > the
> >> > > > > > > > > > > > > > > existing
> >> > > > > > > > > > > > > > > > > > source
> >> > > > > > > > > > > > > > > > > > > > interface. This means we will have
> >> > Pulsar
> >> > > > > > > connector
> >> > > > > > > > > in
> >> > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > 1.10,
> >> > > > > > > > > > > > > > > > > > either
> >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > Because we are going to have
> Pulsar
> >> > sink
> >> > > > and
> >> > > > > > > source
> >> > > > > > > > > > > checked
> >> > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > separately,
> >> > > > > > > > > > > > > > > > > > > > it might make sense to have two
> >> FLIPs,
> >> > > one
> >> > > > > for
> >> > > > > > > > Pulsar
> >> > > > > > > > > > > sink
> >> > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > another
> >> > > > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can start
> the
> >> > work
> >> > > on
> >> > > > > > > Pulsar
> >> > > > > > > > > sink
> >> > > > > > > > > > > > right
> >> > > > > > > > > > > > > > > away.
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM
> Sijie
> >> > Guo
> >> > > <
> >> > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink
> >> community?
> >> > > > Shall
> >> > > > > > we
> >> > > > > > > > wait
> >> > > > > > > > > > for
> >> > > > > > > > > > > > > FLIP-27
> >> > > > > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > > > > > > shall we
> >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And what
> the
> >> > > next
> >> > > > > > steps
> >> > > > > > > > are?
> >> > > > > > > > > > :-)
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > > > > > > Sijie
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM
> >> Bowen
> >> > > Li <
> >> > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > Hi,
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar
> connector
> >> > in
> >> > > > > Flink
> >> > > > > > > can
> >> > > > > > > > > be a
> >> > > > > > > > > > > > good
> >> > > > > > > > > > > > > > > mutual
> >> > > > > > > > > > > > > > > > > > > benefit
> >> > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > both communities.
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > Another perspective is that
> >> Pulsar
> >> > > > > > connector
> >> > > > > > > is
> >> > > > > > > > > the
> >> > > > > > > > > > > 1st
> >> > > > > > > > > > > > > > > > streaming
> >> > > > > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > > > > that integrates with Flink's
> >> > metadata
> >> > > > > > > > management
> >> > > > > > > > > > > system
> >> > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > Catalog
> >> > > > > > > > > > > > > > > > > > > APIs.
> >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the
> >> > > > integration
> >> > > > > > > turns
> >> > > > > > > > > out
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > > > > whether
> >> > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > need to
> >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack,
> >> which
> >> > > are
> >> > > > > > > > currently
> >> > > > > > > > > in
> >> > > > > > > > > > > > > Beta, to
> >> > > > > > > > > > > > > > > > > cater
> >> > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus
> I'm
> >> in
> >> > > > favor
> >> > > > > of
> >> > > > > > > > > merging
> >> > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > into
> >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10.
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller
> >> sized
> >> > > > PRs,
> >> > > > > > e.g.
> >> > > > > > > > > maybe
> >> > > > > > > > > > > one
> >> > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > basic
> >> > > > > > > > > > > > > > > > > > > > > > source/sink functionalities
> and
> >> > > another
> >> > > > > for
> >> > > > > > > > > schema
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > > > > catalog
> >> > > > > > > > > > > > > > > > > > > > > integration,
> >> > > > > > > > > > > > > > > > > > > > > > just to make them easier to
> >> review.
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to
> wait
> >> for
> >> > > > > > FLIP-27.
> >> > > > > > > > But
> >> > > > > > > > > I
> >> > > > > > > > > > > > don't
> >> > > > > > > > > > > > > > > think
> >> > > > > > > > > > > > > > > > > > > FLIP-27
> >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in cases
> >> where
> >> > it
> >> > > > > > cannot
> >> > > > > > > > make
> >> > > > > > > > > > its
> >> > > > > > > > > > > > way
> >> > > > > > > > > > > > > > > into
> >> > > > > > > > > > > > > > > > > 1.10
> >> > > > > > > > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable
> amount
> >> of
> >> > > time
> >> > > > > for
> >> > > > > > > > > > > committers
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > review
> >> > > > > > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully
> adapt
> >> to
> >> > > new
> >> > > > > > > > > interfaces.
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > Bowen
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM
> >> > Becket
> >> > > > > Qin <
> >> > > > > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > Hi Till,
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > You are right. It all
> depends
> >> on
> >> > > when
> >> > > > > the
> >> > > > > > > new
> >> > > > > > > > > > > source
> >> > > > > > > > > > > > > > > > interface
> >> > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > going
> >> > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I
> think it
> >> > > would
> >> > > > > be
> >> > > > > > > > there
> >> > > > > > > > > in
> >> > > > > > > > > > > > > about a
> >> > > > > > > > > > > > > > > > month
> >> > > > > > > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > > > > > > so.
> >> > > > > > > > > > > > > > > > > > > > > > But
> >> > > > > > > > > > > > > > > > > > > > > > > I could be too optimistic.
> It
> >> > would
> >> > > > > also
> >> > > > > > be
> >> > > > > > > > > good
> >> > > > > > > > > > to
> >> > > > > > > > > > > > > hear
> >> > > > > > > > > > > > > > > what
> >> > > > > > > > > > > > > > > > > do
> >> > > > > > > > > > > > > > > > > > > > > Aljoscha
> >> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as they
> are
> >> > also
> >> > > > > > involved
> >> > > > > > > > in
> >> > > > > > > > > > > > FLIP-27.
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > In general I think we should
> >> have
> >> > > > > Pulsar
> >> > > > > > > > > > connector
> >> > > > > > > > > > > in
> >> > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > 1.10,
> >> > > > > > > > > > > > > > > > > > > > > > > preferably with the new
> source
> >> > > > > interface.
> >> > > > > > > We
> >> > > > > > > > > can
> >> > > > > > > > > > > also
> >> > > > > > > > > > > > > check
> >> > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > right
> >> > > > > > > > > > > > > > > > > > > > > > now
> >> > > > > > > > > > > > > > > > > > > > > > > with old source interface,
> but
> >> I
> >> > > > > suspect
> >> > > > > > > few
> >> > > > > > > > > > users
> >> > > > > > > > > > > > > will use
> >> > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > before
> >> > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > next official release.
> >> Therefore,
> >> > > it
> >> > > > > > seems
> >> > > > > > > > > > > reasonable
> >> > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > wait a
> >> > > > > > > > > > > > > > > > > > > little
> >> > > > > > > > > > > > > > > > > > > > > > bit
> >> > > > > > > > > > > > > > > > > > > > > > > to see whether we can jump
> to
> >> the
> >> > > new
> >> > > > > > > source
> >> > > > > > > > > > > > > interface. As
> >> > > > > > > > > > > > > > > > long
> >> > > > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > > > make
> >> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it,
> >> waiting a
> >> > > > > little
> >> > > > > > > bit
> >> > > > > > > > > > > doesn't
> >> > > > > > > > > > > > > seem
> >> > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > hurt
> >> > > > > > > > > > > > > > > > > > > much.
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59
> PM
> >> > Till
> >> > > > > > > Rohrmann
> >> > > > > > > > <
> >> > > > > > > > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the
> >> problem
> >> > > > would
> >> > > > > be
> >> > > > > > > if
> >> > > > > > > > we
> >> > > > > > > > > > > > > committed
> >> > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > connector before the new
> >> source
> >> > > > > > interface
> >> > > > > > > > is
> >> > > > > > > > > > > ready.
> >> > > > > > > > > > > > > If I
> >> > > > > > > > > > > > > > > > > > > understood
> >> > > > > > > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to
> >> > > support
> >> > > > > the
> >> > > > > > > old
> >> > > > > > > > > > source
> >> > > > > > > > > > > > > > > interface
> >> > > > > > > > > > > > > > > > > > > anyway
> >> > > > > > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > existing connectors. By
> >> > checking
> >> > > it
> >> > > > > in
> >> > > > > > > > early
> >> > > > > > > > > I
> >> > > > > > > > > > > > could
> >> > > > > > > > > > > > > see
> >> > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > benefit
> >> > > > > > > > > > > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > > > > > > > > > > our users could start
> using
> >> the
> >> > > > > > connector
> >> > > > > > > > > > > earlier.
> >> > > > > > > > > > > > > > > > Moreover,
> >> > > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar
> >> > > integration
> >> > > > > is
> >> > > > > > > > being
> >> > > > > > > > > > > > delayed
> >> > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > case
> >> > > > > > > > > > > > > > > > > > > that the
> >> > > > > > > > > > > > > > > > > > > > > > > > source interface should be
> >> > > delayed.
> >> > > > > The
> >> > > > > > > > only
> >> > > > > > > > > > > > > downside I
> >> > > > > > > > > > > > > > > see
> >> > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > extra
> >> > > > > > > > > > > > > > > > > > > > > > > > review effort and
> potential
> >> > fixes
> >> > > > > which
> >> > > > > > > > might
> >> > > > > > > > > > be
> >> > > > > > > > > > > > > > > irrelevant
> >> > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > > > > > > source interface
> >> > implementation.
> >> > > I
> >> > > > > > guess
> >> > > > > > > it
> >> > > > > > > > > > > mainly
> >> > > > > > > > > > > > > > > depends
> >> > > > > > > > > > > > > > > > on
> >> > > > > > > > > > > > > > > > > > how
> >> > > > > > > > > > > > > > > > > > > > > > certain
> >> > > > > > > > > > > > > > > > > > > > > > > > we are when the new source
> >> > > > interface
> >> > > > > > will
> >> > > > > > > > be
> >> > > > > > > > > > > ready.
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > > > > > > > > > > > > Till
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> 8:56
> >> AM
> >> > > > Becket
> >> > > > > > > Qin <
> >> > > > > > > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your
> >> > > thoughts.
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some
> >> update
> >> > > on
> >> > > > > > > FLIP-27.
> >> > > > > > > > > > > > Although
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > FLIP
> >> > > > > > > > > > > > > > > > > > > wiki
> >> > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread has
> been
> >> > > quiet
> >> > > > > for
> >> > > > > > > some
> >> > > > > > > > > > > time, a
> >> > > > > > > > > > > > > few
> >> > > > > > > > > > > > > > > > > > > committer /
> >> > > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink
> >> > community
> >> > > > > were
> >> > > > > > > > > actually
> >> > > > > > > > > > > > > > > prototyping
> >> > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > entire
> >> > > > > > > > > > > > > > > > > > > > > > > > thing.
> >> > > > > > > > > > > > > > > > > > > > > > > > > We have made some good
> >> > progress
> >> > > > > there
> >> > > > > > > but
> >> > > > > > > > > > want
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > > > > update
> >> > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > FLIP
> >> > > > > > > > > > > > > > > > > > > > > > wiki
> >> > > > > > > > > > > > > > > > > > > > > > > > > after the entire thing
> is
> >> > > > verified
> >> > > > > to
> >> > > > > > > > work
> >> > > > > > > > > in
> >> > > > > > > > > > > > case
> >> > > > > > > > > > > > > > > there
> >> > > > > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > > some
> >> > > > > > > > > > > > > > > > > > > > > > last
> >> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the
> >> > > > > > implementation.
> >> > > > > > > I
> >> > > > > > > > > > don't
> >> > > > > > > > > > > > > have an
> >> > > > > > > > > > > > > > > > > exact
> >> > > > > > > > > > > > > > > > > > > ETA
> >> > > > > > > > > > > > > > > > > > > > > yet,
> >> > > > > > > > > > > > > > > > > > > > > > > > but I
> >> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be
> >> > within
> >> > > a
> >> > > > > > month
> >> > > > > > > or
> >> > > > > > > > > so.
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the
> >> > > current
> >> > > > > > Flink
> >> > > > > > > > > Pulsar
> >> > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > see if
> >> > > > > > > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It
> would be
> >> > > good
> >> > > > to
> >> > > > > > > avoid
> >> > > > > > > > > the
> >> > > > > > > > > > > > case
> >> > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > checked
> >> > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with
> some
> >> > > review
> >> > > > > > > efforts
> >> > > > > > > > > and
> >> > > > > > > > > > > > > shortly
> >> > > > > > > > > > > > > > > > after
> >> > > > > > > > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is
> ready.
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> 8:39
> >> > AM
> >> > > > > Yijie
> >> > > > > > > > Shen <
> >> > > > > > > > > > > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the
> >> feedback
> >> > > and
> >> > > > > > > > > > suggestions!
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the
> goal
> >> of
> >> > > the
> >> > > > > > > > connector
> >> > > > > > > > > > has
> >> > > > > > > > > > > > > always
> >> > > > > > > > > > > > > > > > been
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > provide
> >> > > > > > > > > > > > > > > > > > > > > > > > > > users with the latest
> >> > > features
> >> > > > of
> >> > > > > > > both
> >> > > > > > > > > > > systems
> >> > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > soon
> >> > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > > > possible.
> >> > > > > > > > > > > > > > > > > > > > > > > We
> >> > > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute
> the
> >> > > > > connector
> >> > > > > > > to
> >> > > > > > > > > > Flink
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > hope
> >> > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > get
> >> > > > > > > > > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and
> feedback
> >> > from
> >> > > > > Flink
> >> > > > > > > > > experts
> >> > > > > > > > > > > to
> >> > > > > > > > > > > > > ensure
> >> > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > high
> >> > > > > > > > > > > > > > > > > > > > > > > quality
> >> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector.
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we
> noticed
> >> its
> >> > > > > > existence
> >> > > > > > > > at
> >> > > > > > > > > > the
> >> > > > > > > > > > > > > > > beginning
> >> > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > reworking
> >> > > > > > > > > > > > > > > > > > > > > > > > > > the connector
> >> > implementation
> >> > > > > based
> >> > > > > > on
> >> > > > > > > > > Flink
> >> > > > > > > > > > > > 1.9;
> >> > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > also
> >> > > > > > > > > > > > > > > > > > > wanted
> >> > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector that
> >> > > supports
> >> > > > > > both
> >> > > > > > > > > batch
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > > stream
> >> > > > > > > > > > > > > > > > > > > computing
> >> > > > > > > > > > > > > > > > > > > > > > based
> >> > > > > > > > > > > > > > > > > > > > > > > > > > on it.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has been
> >> > inactive
> >> > > > for
> >> > > > > > > some
> >> > > > > > > > > > time,
> >> > > > > > > > > > > so
> >> > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > decided
> >> > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > provide
> >> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with most
> of
> >> > the
> >> > > > new
> >> > > > > > > > > features,
> >> > > > > > > > > > > such
> >> > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > type
> >> > > > > > > > > > > > > > > > > > > > > > > system
> >> > > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog
> API
> >> > > first.
> >> > > > We
> >> > > > > > > will
> >> > > > > > > > > pay
> >> > > > > > > > > > > > > attention
> >> > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > progress
> >> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually
> >> and
> >> > > > > > > incorporate
> >> > > > > > > > it
> >> > > > > > > > > > > with
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > > > > soon
> >> > > > > > > > > > > > > > > > > > > > > > > > > > as possible.
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test
> status
> >> > of
> >> > > > the
> >> > > > > > > > > connector,
> >> > > > > > > > > > > we
> >> > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > following
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > other
> >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in
> Flink
> >> > > > > > repository
> >> > > > > > > > and
> >> > > > > > > > > > > aimed
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > provide
> >> > > > > > > > > > > > > > > > > > > > > > throughout
> >> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We
> are
> >> > > also
> >> > > > > > happy
> >> > > > > > > to
> >> > > > > > > > > > hear
> >> > > > > > > > > > > > > > > > suggestions
> >> > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from the
> >> Flink
> >> > > > > > community
> >> > > > > > > to
> >> > > > > > > > > > > improve
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > stability and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the
> >> > connector
> >> > > > > > > > > continuously.
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> >> 5:59
> >> > > AM
> >> > > > > > Sijie
> >> > > > > > > > Guo
> >> > > > > > > > > <
> >> > > > > > > > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for
> the
> >> > > > > comments
> >> > > > > > > and
> >> > > > > > > > > > > > feedback.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that
> the
> >> > > main
> >> > > > > > > question
> >> > > > > > > > > > here
> >> > > > > > > > > > > is
> >> > > > > > > > > > > > > > > about -
> >> > > > > > > > > > > > > > > > > > "how
> >> > > > > > > > > > > > > > > > > > > can
> >> > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > community maintain
> the
> >> > > > > > connector?".
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two
> thoughts
> >> > from
> >> > > > > > myself.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and
> >> where
> >> > to
> >> > > > > host
> >> > > > > > > this
> >> > > > > > > > > > > > > integration
> >> > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > kind
> >> > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > less
> >> > > > > > > > > > > > > > > > > > > > > > > > > > important
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe
> there
> >> can
> >> > > be
> >> > > > > many
> >> > > > > > > > ways
> >> > > > > > > > > to
> >> > > > > > > > > > > > > achieve
> >> > > > > > > > > > > > > > > it.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the
> >> > > contribution,
> >> > > > > what
> >> > > > > > > we
> >> > > > > > > > > are
> >> > > > > > > > > > > > > looking
> >> > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > here
> >> > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > how
> >> > > > > > > > > > > > > > > > > > > > > > > > these
> >> > > > > > > > > > > > > > > > > > > > > > > > > > two
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can
> build
> >> the
> >> > > > > > > > collaboration
> >> > > > > > > > > > > > > > > relationship
> >> > > > > > > > > > > > > > > > on
> >> > > > > > > > > > > > > > > > > > > > > > developing
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration
> between
> >> > > > Pulsar
> >> > > > > > and
> >> > > > > > > > > Flink.
> >> > > > > > > > > > > > Even
> >> > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > can
> >> > > > > > > > > > > > > > > > > try
> >> > > > > > > > > > > > > > > > > > > our
> >> > > > > > > > > > > > > > > > > > > > > > best
> >> > > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > catch
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates
> in
> >> > Flink
> >> > > > > > > > community.
> >> > > > > > > > > We
> >> > > > > > > > > > > are
> >> > > > > > > > > > > > > still
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact
> that we
> >> > > have
> >> > > > > less
> >> > > > > > > > > > > experiences
> >> > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > than
> >> > > > > > > > > > > > > > > > > > > > > folks
> >> > > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In order
> to
> >> > make
> >> > > > > sure
> >> > > > > > we
> >> > > > > > > > > > > maintain
> >> > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > deliver
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality
> >> > pulsar-flink
> >> > > > > > > > integration
> >> > > > > > > > > > to
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > users
> >> > > > > > > > > > > > > > > > > who
> >> > > > > > > > > > > > > > > > > > > use
> >> > > > > > > > > > > > > > > > > > > > > both
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we
> need
> >> > some
> >> > > > help
> >> > > > > > > from
> >> > > > > > > > > the
> >> > > > > > > > > > > > > experts
> >> > > > > > > > > > > > > > > from
> >> > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > community.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been
> >> following
> >> > > > > FLIP-27
> >> > > > > > > > for a
> >> > > > > > > > > > > > while.
> >> > > > > > > > > > > > > > > > > Originally
> >> > > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > > were
> >> > > > > > > > > > > > > > > > > > > > > > > > > > thinking
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the
> >> > > > connectors
> >> > > > > > back
> >> > > > > > > > > after
> >> > > > > > > > > > > > > > > integrating
> >> > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced
> in
> >> > > > FLIP-27.
> >> > > > > > But
> >> > > > > > > we
> >> > > > > > > > > > > decided
> >> > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > initiate
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > conversation
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as
> possible.
> >> > > Because
> >> > > > > we
> >> > > > > > > > > believe
> >> > > > > > > > > > > > there
> >> > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > > > > > > benefits
> >> > > > > > > > > > > > > > > > > > > > > > > > doing
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than
> >> later.
> >> > > As
> >> > > > > part
> >> > > > > > > of
> >> > > > > > > > > > > > > contribution,
> >> > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > can
> >> > > > > > > > > > > > > > > > > > > help
> >> > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > community understand
> >> more
> >> > > > about
> >> > > > > > > > Pulsar
> >> > > > > > > > > > and
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > potential
> >> > > > > > > > > > > > > > > > > > > > > > > integration
> >> > > > > > > > > > > > > > > > > > > > > > > > > > points.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also
> help
> >> > Flink
> >> > > > > > > community
> >> > > > > > > > > > > verify
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > > > > API
> >> > > > > > > > > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > > > > > > > > well
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API
> (e.g.
> >> > > > catalog
> >> > > > > > > API).
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019
> at
> >> > 5:24
> >> > > > AM
> >> > > > > > > Becket
> >> > > > > > > > > > Qin <
> >> > > > > > > > > > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> >> interest
> >> > > in
> >> > > > > > > > > contributing
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > connector.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I
> think
> >> > > having
> >> > > > > > Pulsar
> >> > > > > > > > > > > connector
> >> > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > strong
> >> > > > > > > > > > > > > > > > > > > > > support
> >> > > > > > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > > > > a
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition
> to
> >> > > Flink.
> >> > > > > So
> >> > > > > > I
> >> > > > > > > am
> >> > > > > > > > > > happy
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > shepherd
> >> > > > > > > > > > > > > > > > > > > this
> >> > > > > > > > > > > > > > > > > > > > > > > effort.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would
> >> also
> >> > > > like
> >> > > > > to
> >> > > > > > > > > provide
> >> > > > > > > > > > > > some
> >> > > > > > > > > > > > > > > > context
> >> > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > recent
> >> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink
> connectors
> >> > > > > ecosystem.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way
> Flink
> >> > > > > maintains
> >> > > > > > > its
> >> > > > > > > > > > > > > connector has
> >> > > > > > > > > > > > > > > > hit
> >> > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > scalability
> >> > > > > > > > > > > > > > > > > > > > > > > > > > bar.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more
> >> > > > connectors
> >> > > > > > > > coming
> >> > > > > > > > > > into
> >> > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > repo,
> >> > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > > > > > > > facing a
> >> > > > > > > > > > > > > > > > > > > > > > > > > > few
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as
> long
> >> > > build
> >> > > > > and
> >> > > > > > > > > testing
> >> > > > > > > > > > > > > time. To
> >> > > > > > > > > > > > > > > > > > address
> >> > > > > > > > > > > > > > > > > > > this
> >> > > > > > > > > > > > > > > > > > > > > > > > > problem,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to
> do
> >> > the
> >> > > > > > > following:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the
> >> > > connectors
> >> > > > > > into
> >> > > > > > > a
> >> > > > > > > > > > > separate
> >> > > > > > > > > > > > > > > > > repository.
> >> > > > > > > > > > > > > > > > > > > This
> >> > > > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to
> >> > potential
> >> > > > > > solution
> >> > > > > > > > to
> >> > > > > > > > > > > > shorten
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > build
> >> > > > > > > > > > > > > > > > > > > time.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the
> >> > > connectors
> >> > > > > to
> >> > > > > > > stay
> >> > > > > > > > > as
> >> > > > > > > > > > > > > ecosystem
> >> > > > > > > > > > > > > > > > > > project
> >> > > > > > > > > > > > > > > > > > > > > while
> >> > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > tries
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good
> >> support
> >> > > for
> >> > > > > > > > > > functionality
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > compatibility
> >> > > > > > > > > > > > > > > > > > > > > > > tests.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Robert
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to
> create
> >> a
> >> > > > Flink
> >> > > > > > > > > Ecosystem
> >> > > > > > > > > > > > > project
> >> > > > > > > > > > > > > > > > > website
> >> > > > > > > > > > > > > > > > > > > and it
> >> > > > > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > > > > > going
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some final
> >> > > approval
> >> > > > > > > > process.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above
> >> > efforts,
> >> > > it
> >> > > > > > would
> >> > > > > > > > be
> >> > > > > > > > > > > great
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > first
> >> > > > > > > > > > > > > > > > > see
> >> > > > > > > > > > > > > > > > > > > if we
> >> > > > > > > > > > > > > > > > > > > > > > can
> >> > > > > > > > > > > > > > > > > > > > > > > > > have
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector
> as
> >> an
> >> > > > > > ecosystem
> >> > > > > > > > > > project
> >> > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > great
> >> > > > > > > > > > > > > > > > > > > support.
> >> > > > > > > > > > > > > > > > > > > > > It
> >> > > > > > > > > > > > > > > > > > > > > > > > would
> >> > > > > > > > > > > > > > > > > > > > > > > > > > be
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how
> the
> >> > > Flink
> >> > > > > > Pulsar
> >> > > > > > > > > > > connector
> >> > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > tested
> >> > > > > > > > > > > > > > > > > > > > > currently
> >> > > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > see
> >> > > > > > > > > > > > > > > > > > > > > > > > > > if
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn
> >> something
> >> > to
> >> > > > > > > maintain
> >> > > > > > > > it
> >> > > > > > > > > > as
> >> > > > > > > > > > > an
> >> > > > > > > > > > > > > > > > ecosystem
> >> > > > > > > > > > > > > > > > > > > project
> >> > > > > > > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > > > > > > > good
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test
> >> > > coverage.
> >> > > > If
> >> > > > > > the
> >> > > > > > > > > > quality
> >> > > > > > > > > > > > as
> >> > > > > > > > > > > > > an
> >> > > > > > > > > > > > > > > > > > ecosystem
> >> > > > > > > > > > > > > > > > > > > > > > project
> >> > > > > > > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > > > > > > hard
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we
> may
> >> as
> >> > > > well
> >> > > > > > > adopt
> >> > > > > > > > it
> >> > > > > > > > > > > into
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > main
> >> > > > > > > > > > > > > > > > > > repo.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another
> ongoing
> >> > > effort
> >> > > > > is
> >> > > > > > > > > FLIP-27
> >> > > > > > > > > > > > where
> >> > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > > making
> >> > > > > > > > > > > > > > > > > > > > > > > changes
> >> > > > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source
> >> connector
> >> > > > > > > architecture
> >> > > > > > > > > and
> >> > > > > > > > > > > > > > > interface.
> >> > > > > > > > > > > > > > > > > This
> >> > > > > > > > > > > > > > > > > > > > > change
> >> > > > > > > > > > > > > > > > > > > > > > > will
> >> > > > > > > > > > > > > > > > > > > > > > > > > > likely
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10.
> >> Therefore
> >> > > > > timing
> >> > > > > > > > wise,
> >> > > > > > > > > if
> >> > > > > > > > > > > we
> >> > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > going
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > have
> >> > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main
> >> > repo, I
> >> > > > am
> >> > > > > > > > > wondering
> >> > > > > > > > > > if
> >> > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > should
> >> > > > > > > > > > > > > > > > > > hold
> >> > > > > > > > > > > > > > > > > > > a
> >> > > > > > > > > > > > > > > > > > > > > > little
> >> > > > > > > > > > > > > > > > > > > > > > > > bit
> >> > > > > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar
> >> > connector
> >> > > > > adapt
> >> > > > > > to
> >> > > > > > > > the
> >> > > > > > > > > > new
> >> > > > > > > > > > > > > > > interface
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > avoid
> >> > > > > > > > > > > > > > > > > > > > > > > > shortly
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket)
> Qin
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> 2019
> >> at
> >> > > 4:32
> >> > > > > PM
> >> > > > > > > > > Chesnay
> >> > > > > > > > > > > > > Schepler <
> >> > > > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite
> worried
> >> > that
> >> > > we
> >> > > > > may
> >> > > > > > > end
> >> > > > > > > > > up
> >> > > > > > > > > > > > > repeating
> >> > > > > > > > > > > > > > > > > > history.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were
> already
> >> 2
> >> > > > > attempts
> >> > > > > > > at
> >> > > > > > > > > > > > > contributing a
> >> > > > > > > > > > > > > > > > > > pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > connector,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > both
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed
> >> > because
> >> > > > no
> >> > > > > > > > > committer
> >> > > > > > > > > > > was
> >> > > > > > > > > > > > > > > getting
> >> > > > > > > > > > > > > > > > > > > involved,
> >> > > > > > > > > > > > > > > > > > > > > > > > despite
> >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor
> >> opening a
> >> > > > > > dedicated
> >> > > > > > > > > > > > discussion
> >> > > > > > > > > > > > > > > thread
> >> > > > > > > > > > > > > > > > > > > about the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > contribution
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and
> >> > getting
> >> > > > > > several
> >> > > > > > > > +1's
> >> > > > > > > > > > > from
> >> > > > > > > > > > > > > > > > > committers.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really
> >> make
> >> > > > sure
> >> > > > > > that
> >> > > > > > > > if
> >> > > > > > > > > we
> >> > > > > > > > > > > > > > > > > welcome/approve
> >> > > > > > > > > > > > > > > > > > > such
> >> > > > > > > > > > > > > > > > > > > > > a
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it
> >> will
> >> > > > > actually
> >> > > > > > > get
> >> > > > > > > > > the
> >> > > > > > > > > > > > > attention
> >> > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > > > deserves.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm
> >> inclined
> >> > > to
> >> > > > > > > > recommend
> >> > > > > > > > > > > > > maintaining
> >> > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > > > > > > > outside
> >> > > > > > > > > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could
> >> link
> >> > to
> >> > > > it
> >> > > > > > from
> >> > > > > > > > the
> >> > > > > > > > > > > > > > > documentation
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > give
> >> > > > > > > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the
> upcoming
> >> > page
> >> > > > for
> >> > > > > > > > sharing
> >> > > > > > > > > > > > > artifacts
> >> > > > > > > > > > > > > > > > among
> >> > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > community
> >> > > > > > > > > > > > > > > > > > > > > > > > > > (what's
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of
> that
> >> > > > anyway?),
> >> > > > > > > this
> >> > > > > > > > > may
> >> > > > > > > > > > > be a
> >> > > > > > > > > > > > > > > better
> >> > > > > > > > > > > > > > > > > > > option.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019
> >> 10:16,
> >> > > Till
> >> > > > > > > > Rohrmann
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot
> for
> >> > > > starting
> >> > > > > > > this
> >> > > > > > > > > > > > discussion
> >> > > > > > > > > > > > > > > > Yijie.
> >> > > > > > > > > > > > > > > > > I
> >> > > > > > > > > > > > > > > > > > > think
> >> > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector
> would
> >> be
> >> > a
> >> > > > very
> >> > > > > > > > > valuable
> >> > > > > > > > > > > > > addition
> >> > > > > > > > > > > > > > > > since
> >> > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > becomes
> >> > > > > > > > > > > > > > > > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular
> and
> >> it
> >> > > > would
> >> > > > > > > > further
> >> > > > > > > > > > > > expand
> >> > > > > > > > > > > > > > > > Flink's
> >> > > > > > > > > > > > > > > > > > > > > > > > > interoperability.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Also
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project
> >> > > > > perspective
> >> > > > > > it
> >> > > > > > > > > makes
> >> > > > > > > > > > > > > sense for
> >> > > > > > > > > > > > > > > > me
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > place
> >> > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream
> >> > > project.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> >> > > > concern/question
> >> > > > > is
> >> > > > > > > how
> >> > > > > > > > > can
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > community
> >> > > > > > > > > > > > > > > > > > > > > > > > maintain
> >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We
> >> have
> >> > > seen
> >> > > > > in
> >> > > > > > > the
> >> > > > > > > > > past
> >> > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > > > connectors
> >> > > > > > > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > > > > > some
> >> > > > > > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > most
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively
> >> developed
> >> > > > > > components
> >> > > > > > > > > > because
> >> > > > > > > > > > > > > they
> >> > > > > > > > > > > > > > > need
> >> > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > be
> >> > > > > > > > > > > > > > > > > > > > > kept
> >> > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > > sync
> >> > > > > > > > > > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external
> system
> >> and
> >> > > > with
> >> > > > > > > Flink.
> >> > > > > > > > > > Given
> >> > > > > > > > > > > > > that
> >> > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > community
> >> > > > > > > > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
> >> > > > maintaining,
> >> > > > > > > > > improving
> >> > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > evolving
> >> > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > connector,
> >> > > > > > > > > > > > > > > > > > > > > > > > > I'm
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic
> that
> >> we
> >> > > can
> >> > > > > > > achieve
> >> > > > > > > > > > this.
> >> > > > > > > > > > > > > Hence,
> >> > > > > > > > > > > > > > > +1
> >> > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > > > > contributing
> >> > > > > > > > > > > > > > > > > > > > > > > > > it
> >> > > > > > > > > > > > > > > > > > > > > > > > > > back
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> >> 2019
> >> > > at
> >> > > > > 2:03
> >> > > > > > > AM
> >> > > > > > > > > > Sijie
> >> > > > > > > > > > > > Guo
> >> > > > > > > > > > > > > <
> >> > > > > > > > > > > > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was
> the
> >> > main
> >> > > > > > driver
> >> > > > > > > > > behind
> >> > > > > > > > > > > > > > > FLINK-9641
> >> > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168,
> >> > > > > > > > > > > > > > > > > > > > > > > > > let
> >> > > > > > > > > > > > > > > > > > > > > > > > > > me
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more
> context
> >> > on
> >> > > > > this.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641
> and
> >> > > > > FLINK-9168
> >> > > > > > > was
> >> > > > > > > > > > > created
> >> > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > bringing
> >> > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > > > > > > > > source
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for
> Flink.
> >> > The
> >> > > > > > > > integration
> >> > > > > > > > > > was
> >> > > > > > > > > > > > done
> >> > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > 1.6.0.
> >> > > > > > > > > > > > > > > > > > > > > > > We
> >> > > > > > > > > > > > > > > > > > > > > > > > > > sent out
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests
> about a
> >> > > year
> >> > > > > ago
> >> > > > > > > and
> >> > > > > > > > we
> >> > > > > > > > > > > ended
> >> > > > > > > > > > > > > up
> >> > > > > > > > > > > > > > > > > > > maintaining
> >> > > > > > > > > > > > > > > > > > > > > > those
> >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for
> >> Pulsar
> >> > > > users
> >> > > > > to
> >> > > > > > > use
> >> > > > > > > > > > Flink
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > process
> >> > > > > > > > > > > > > > > > > > > event
> >> > > > > > > > > > > > > > > > > > > > > > > streams
> >> > > > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > ).
> >> > > > > > > > > > > > > > > > > > > > > > > > > > The
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6
> integration
> >> is
> >> > > > > pretty
> >> > > > > > > > simple
> >> > > > > > > > > > and
> >> > > > > > > > > > > > > there
> >> > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > no
> >> > > > > > > > > > > > > > > > > > > schema
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past
> >> year,
> >> > we
> >> > > > > have
> >> > > > > > > > made a
> >> > > > > > > > > > lot
> >> > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > changes
> >> > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > brought
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar
> schema as
> >> > the
> >> > > > > > > > first-class
> >> > > > > > > > > > > > > citizen in
> >> > > > > > > > > > > > > > > > > > Pulsar.
> >> > > > > > > > > > > > > > > > > > > We
> >> > > > > > > > > > > > > > > > > > > > > > also
> >> > > > > > > > > > > > > > > > > > > > > > > > > > integrated
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other
> computing
> >> > > > engines
> >> > > > > > for
> >> > > > > > > > > > > processing
> >> > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > event
> >> > > > > > > > > > > > > > > > > > > > > > streams
> >> > > > > > > > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to
> >> > rethink
> >> > > > how
> >> > > > > > to
> >> > > > > > > > > > > integrate
> >> > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > best
> >> > > > > > > > > > > > > > > > > > > > > > > > > way.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Then
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement
> the
> >> > > > > > pulsar-flink
> >> > > > > > > > > > > > connectors
> >> > > > > > > > > > > > > from
> >> > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > ground
> >> > > > > > > > > > > > > > > > > > > > > up
> >> > > > > > > > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > > > > > > > > schema
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table
> API
> >> > and
> >> > > > > > catalog
> >> > > > > > > > API
> >> > > > > > > > > as
> >> > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > first-class
> >> > > > > > > > > > > > > > > > > > > > > citizen
> >> > > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration.
> >> With
> >> > > that
> >> > > > > > being
> >> > > > > > > > > said,
> >> > > > > > > > > > > in
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > > > pulsar-flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can
> register
> >> > > > pulsar
> >> > > > > > as a
> >> > > > > > > > > flink
> >> > > > > > > > > > > > > catalog
> >> > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > query /
> >> > > > > > > > > > > > > > > > > > > > > > > process
> >> > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > event
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using
> >> > Flink
> >> > > > SQL.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an
> >> example
> >> > > > about
> >> > > > > > how
> >> > > > > > > > to
> >> > > > > > > > > > use
> >> > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > a
> >> > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > catalog:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has
> also
> >> > > > written a
> >> > > > > > > blog
> >> > > > > > > > > post
> >> > > > > > > > > > > > > > > explaining
> >> > > > > > > > > > > > > > > > > why
> >> > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > > > > > > re-implement
> >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector
> with
> >> > Flink
> >> > > > 1.9
> >> > > > > > and
> >> > > > > > > > > what
> >> > > > > > > > > > > are
> >> > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > changes
> >> > > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > > > made
> >> > > > > > > > > > > > > > > > > > > > > > in
> >> > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > new
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe
> >> Pulsar
> >> > is
> >> > > > not
> >> > > > > > > just
> >> > > > > > > > a
> >> > > > > > > > > > > simple
> >> > > > > > > > > > > > > data
> >> > > > > > > > > > > > > > > > sink
> >> > > > > > > > > > > > > > > > > > or
> >> > > > > > > > > > > > > > > > > > > > > source
> >> > > > > > > > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > It
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can
> be
> >> a
> >> > > > fully
> >> > > > > > > > > integrated
> >> > > > > > > > > > > > > streaming
> >> > > > > > > > > > > > > > > > > data
> >> > > > > > > > > > > > > > > > > > > > > storage
> >> > > > > > > > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink,
> >> > source,
> >> > > > > > > > > > schema/catalog
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > state).
> >> > > > > > > > > > > > > > > > > > The
> >> > > > > > > > > > > > > > > > > > > > > > > > combination
> >> > > > > > > > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar
> can
> >> > > create
> >> > > > a
> >> > > > > > > great
> >> > > > > > > > > > > > streaming
> >> > > > > > > > > > > > > > > > > warehouse
> >> > > > > > > > > > > > > > > > > > > > > > > architecture
> >> > > > > > > > > > > > > > > > > > > > > > > > > for
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> streaming-first,
> >> > > > unified
> >> > > > > > > data
> >> > > > > > > > > > > > > processing.
> >> > > > > > > > > > > > > > > > Since
> >> > > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > > > > > > talking
> >> > > > > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute
> >> Pulsar
> >> > > > > > > integration
> >> > > > > > > > to
> >> > > > > > > > > > > Flink
> >> > > > > > > > > > > > > here,
> >> > > > > > > > > > > > > > > > we
> >> > > > > > > > > > > > > > > > > > are
> >> > > > > > > > > > > > > > > > > > > also
> >> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain,
> >> improve
> >> > > and
> >> > > > > > evolve
> >> > > > > > > > the
> >> > > > > > > > > > > > > integration
> >> > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > help
> >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both
> >> Flink
> >> > > and
> >> > > > > > > Pulsar.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this
> give
> >> > you a
> >> > > > bit
> >> > > > > > > more
> >> > > > > > > > > > > > background
> >> > > > > > > > > > > > > > > about
> >> > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration.
> Let
> >> > me
> >> > > > know
> >> > > > > > > what
> >> > > > > > > > > are
> >> > > > > > > > > > > your
> >> > > > > > > > > > > > > > > > thoughts.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep
> 3,
> >> > 2019
> >> > > at
> >> > > > > > 11:54
> >> > > > > > > > AM
> >> > > > > > > > > > Yun
> >> > > > > > > > > > > > > Tang <
> >> > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see
> that
> >> > > Pulsar
> >> > > > > > > becomes
> >> > > > > > > > > > more
> >> > > > > > > > > > > > and
> >> > > > > > > > > > > > > more
> >> > > > > > > > > > > > > > > > > > popular
> >> > > > > > > > > > > > > > > > > > > > > > > recently
> >> > > > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > very
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more
> >> > people
> >> > > > > > willing
> >> > > > > > > to
> >> > > > > > > > > > > > > contribute to
> >> > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > ecosystem.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any
> >> > further
> >> > > > > > > > discussion,
> >> > > > > > > > > > > would
> >> > > > > > > > > > > > > you
> >> > > > > > > > > > > > > > > > please
> >> > > > > > > > > > > > > > > > > > > give
> >> > > > > > > > > > > > > > > > > > > > > some
> >> > > > > > > > > > > > > > > > > > > > > > > > > > explanation
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the
> >> relationship
> >> > > > > between
> >> > > > > > > this
> >> > > > > > > > > > > thread
> >> > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > current
> >> > > > > > > > > > > > > > > > > > > > > existing
> >> > > > > > > > > > > > > > > > > > > > > > > > JIRAs
> >> > > > > > > > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1]
> and
> >> > sink
> >> > > > [2]
> >> > > > > > > > > > connector?
> >> > > > > > > > > > > > > Will the
> >> > > > > > > > > > > > > > > > > > > > > contribution
> >> > > > > > > > > > > > > > > > > > > > > > > > > contains
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > part
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or
> >> > > totally
> >> > > > > > > > different
> >> > > > > > > > > > > > > > > > implementation?
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > https://issues.apache.org/jira/browse/FLINK-9641
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > https://issues.apache.org/jira/browse/FLINK-9168
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > ________________________________
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie
> >> Shen
> >> > <
> >> > > > > > > > > > > > > > > [hidden email]>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent:
> Tuesday,
> >> > > > > September
> >> > > > > > 3,
> >> > > > > > > > > 2019
> >> > > > > > > > > > > > 13:57
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> >> > > > > [hidden email]
> >> > > > > > <
> >> > > > > > > > > > > > > > > > [hidden email]
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject:
> >> > [DISCUSS]
> >> > > > > > > Contribute
> >> > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > > > back
> >> > > > > > > > > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink
> >> > > Community!
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would
> like to
> >> > > open
> >> > > > > the
> >> > > > > > > > > > discussion
> >> > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > contributing
> >> > > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector
> [0]
> >> > back
> >> > > to
> >> > > > > > > Flink.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
> >> > > > introduction
> >> > > > > > to
> >> > > > > > > > > Apache
> >> > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache
> >> Pulsar[1]
> >> > > is a
> >> > > > > > > > > > multi-tenant,
> >> > > > > > > > > > > > > > > > > > > high-performance
> >> > > > > > > > > > > > > > > > > > > > > > > > > distributed
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub
> >> messaging
> >> > > > > system.
> >> > > > > > > > > Pulsar
> >> > > > > > > > > > > > > includes
> >> > > > > > > > > > > > > > > > > > multiple
> >> > > > > > > > > > > > > > > > > > > > > > features
> >> > > > > > > > > > > > > > > > > > > > > > > > > such
> >> > > > > > > > > > > > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native
> support
> >> > for
> >> > > > > > multiple
> >> > > > > > > > > > > clusters
> >> > > > > > > > > > > > > in a
> >> > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > instance,
> >> > > > > > > > > > > > > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> >> > > > > geo-replication
> >> > > > > > of
> >> > > > > > > > > > > messages
> >> > > > > > > > > > > > > across
> >> > > > > > > > > > > > > > > > > > > clusters,
> >> > > > > > > > > > > > > > > > > > > > > > very
> >> > > > > > > > > > > > > > > > > > > > > > > > low
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and
> end-to-end
> >> > > > latency,
> >> > > > > > > > > seamless
> >> > > > > > > > > > > > > > > scalability
> >> > > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > over a
> >> > > > > > > > > > > > > > > > > > > > > > > > million
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and
> guaranteed
> >> > > > message
> >> > > > > > > > delivery
> >> > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > persistent
> >> > > > > > > > > > > > > > > > > > > message
> >> > > > > > > > > > > > > > > > > > > > > > > > storage
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by
> >> > Apache
> >> > > > > > > > BookKeeper.
> >> > > > > > > > > > > > > Nowadays,
> >> > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > has
> >> > > > > > > > > > > > > > > > > > > > > been
> >> > > > > > > > > > > > > > > > > > > > > > > > > adopted
> >> > > > > > > > > > > > > > > > > > > > > > > > > > by
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and
> more
> >> > > > > > companies[2].
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The
> status
> >> of
> >> > > > Pulsar
> >> > > > > > > Flink
> >> > > > > > > > > > > > connector
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar
> >> Flink
> >> > > > > > connector
> >> > > > > > > we
> >> > > > > > > > > are
> >> > > > > > > > > > > > > planning
> >> > > > > > > > > > > > > > > to
> >> > > > > > > > > > > > > > > > > > > > > contribute
> >> > > > > > > > > > > > > > > > > > > > > > is
> >> > > > > > > > > > > > > > > > > > > > > > > > > built
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0
> and
> >> > > > Pulsar
> >> > > > > > > 2.4.0.
> >> > > > > > > > > The
> >> > > > > > > > > > > > main
> >> > > > > > > > > > > > > > > > features
> >> > > > > > > > > > > > > > > > > > > are:
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as
> a
> >> > > > streaming
> >> > > > > > > > source
> >> > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > exactly-once
> >> > > > > > > > > > > > > > > > > > > > > > guarantee.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink
> >> streaming
> >> > > > > results
> >> > > > > > to
> >> > > > > > > > > > Pulsar
> >> > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > > > > > at-least-once
> >> > > > > > > > > > > > > > > > > > > > > > > > > semantics.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > (We
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update
> >> this
> >> > > to
> >> > > > > > > > > exactly-once
> >> > > > > > > > > > > as
> >> > > > > > > > > > > > > well
> >> > > > > > > > > > > > > > > > when
> >> > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > gets
> >> > > > > > > > > > > > > > > > > > > > > > > > all
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction
> >> > > features
> >> > > > > > ready
> >> > > > > > > in
> >> > > > > > > > > its
> >> > > > > > > > > > > > 2.5.0
> >> > > > > > > > > > > > > > > > > version)
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon
> >> > Flink
> >> > > > new
> >> > > > > > > Table
> >> > > > > > > > > API
> >> > > > > > > > > > > Type
> >> > > > > > > > > > > > > > > system
> >> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> >> > > > > > > > > > > > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > > > > > > > > > > > > can
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> automatically
> >> > > > > > (de)serialize
> >> > > > > > > > > > > messages
> >> > > > > > > > > > > > > with
> >> > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > help
> >> > > > > > > > > > > > > > > > > > > of
> >> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> >> > > > > > > > > > > > > > > > > > > > > > > > > > schema.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate
> >> with
> >> > > > Flink
> >> > > > > > new
> >> > > > > > > > > > Catalog
> >> > > > > > > > > > > > API
> >> > > > > > > > > > > > > > > > > > > (FLIP-30[4]),
> >> > > > > > > > > > > > > > > > > > > > > > which
> >> > > > > > > > > > > > > > > > > > > > > > > > > > enables
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of
> Pulsar
> >> > > topics
> >> > > > as
> >> > > > > > > > tables
> >> > > > > > > > > in
> >> > > > > > > > > > > > > Table API
> >> > > > > > > > > > > > > > > > as
> >> > > > > > > > > > > > > > > > > > > well as
> >> > > > > > > > > > > > > > > > > > > > > > SQL
> >> > > > > > > > > > > > > > > > > > > > > > > > > > client.
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> >> > > > > > > > > > > > > > > > >
> >> https://github.com/streamnative/pulsar-flink
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> >> > > > > > > > https://pulsar.apache.org/
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> >> > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>

[Becket Qin]

Hi Till,

My understanding is that we are going to hold back a little bit on the
Pulsar source connector until FLIP-27 is ready. If we check in the old
Pulsar source connector right away, we will have to maintain that old
connector for quite a while (a year or more), which is something we don't
want to do.

So the conclusion is following:
- We will try our best to check in both FLIP-27 and the Pulsar source
connector in 1.10.
- In case there isn't enough time to get Pulsar source connector checked in
1.10, we will link prominently to the external Pulsar connector repo in our
connector doc so that users can find that easily.

BTW, I think we can start the work on Pulsar sink connector given that
there is no clear plan on whether or how we are going to change the sink
interface.

@Sijie, @Yijie, what do you think?

Thanks,

Jiangjie (Becket) Qin

On Tue, Sep 24, 2019 at 4:40 PM Till Rohrmann <[hidden email]> wrote:

> What is the conclusion of this discussion? Are we gonna merge the old
> source/sink or do we believe that FLIP-27 will be completed in time so that
> we can merge the new source right away?
>
> Cheers,
> Till
>
> On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <[hidden email]> wrote:
>
> >
> >
> > unscribe
> >
> >
> >
> >
> >
> >
> > At 2019-09-24 08:53:25, "Becket Qin" <[hidden email]> wrote:
> > >Thanks, Stephan.
> > >
> > >Sounds good to me. We can still try our best to get new Pulsar connector
> > in
> > >Flink 1.10. In case we do not have time to do that, we will prominently
> > >link the Pulsar connector from the Flink connector docs.
> > >
> > >Thanks,
> > >
> > >Jiangjie (Becket) Qin
> > >
> > >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <[hidden email]> wrote:
> > >
> > >> Okay, I see your point, Becket.
> > >>
> > >> Then let us prominently link the Pulsar connector from the Flink
> > connector
> > >> docs then, so that users can find it easily.
> > >>
> > >> As soon as FLIP 27 is done, we reach out the Pulsar folks to
> contribute
> > a
> > >> new connector.
> > >>
> > >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]>
> > wrote:
> > >>
> > >> > Hi Stephan,
> > >> >
> > >> > I have no doubt about the value of adding Pulsar connector to Flink
> > repo.
> > >> > My concern is about how exactly we are going to do it.
> > >> >
> > >> > As mentioned before, I believe that we can handle connectors more
> > >> > > pragmatically and less strict than the core of Flink, if it helps
> > >> > unlocking
> > >> > > users.
> > >> >
> > >> > I can see the benefit of being less restrict for the initial
> connector
> > >> code
> > >> > adoption. However, I don't think we should be less restrict on the
> > >> > maintenance commitment once the code is in Flink repo. It only makes
> > >> sense
> > >> > to check in something and ask users to use if we plan to maintain
> it.
> > >> >
> > >> > If I understand correctly, the current plan so far is following:
> > >> > 1. release 1.10
> > >> >    - Check in Pulsar connector on old interface and label it as beta
> > >> > version.
> > >> >    - encourage users to try it and report bugs.
> > >> > 2. release 1.11
> > >> >    - Check in Pulsar connector on new interface (a.k.a new Pulsar
> > >> > connector) and label it as beta version
> > >> >    - Deprecate the old Pulsar connector
> > >> >    - Fix bugs reported on old Pulsar connector from release 1.10
> > >> >    - Ask users to migrate from old Pulsar connector to new Pulsar
> > >> connector
> > >> > 3. release 1.12
> > >> >    - Announce end of support for old Pulsar connector and remove the
> > code
> > >> >    - Fix bugs reported on new Pulsar connector.
> > >> >
> > >> > If this is the plan, it seems neither Flink nor the users trying the
> > old
> > >> > Pulsar connector will benefit from this experimental old Pulsar
> > >> connector,
> > >> > because whatever feedbacks we got or bugs we fix on the old Pulsar
> > >> > connector are immediately thrown away in one or two releases.
> > >> >
> > >> > If we check in the old Pulsar connector right now, the only option I
> > see
> > >> is
> > >> > to maintain it for a while (e.g. a year or more). IMO, the immediate
> > >> > deprecation and code removal hurts the users much more than asking
> > them
> > >> to
> > >> > wait for another release. I personally think that we can avoid this
> > >> > maintenance burden by going directly to the new Pulsar connector,
> > >> > especially given that users can still use the connector even if they
> > are
> > >> > not in Flink repo. That said, I am OK with maintaining both old and
> > new
> > >> > Pulsar connector if we believe that having the Pulsar connector
> > available
> > >> > right now in Flink repo is more important.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jiangjie (Becket) Qin
> > >> >
> > >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]>
> > wrote:
> > >> >
> > >> > > My assumption is as Sijie's, that once the connector is either
> part
> > of
> > >> > > Flink, or part of the streamnative repo. No double maintenance.
> > >> > >
> > >> > > I feel this discussion is very much caught in problems that are
> all
> > >> > > solvable if we want to solve them.
> > >> > > Maybe we can think what our goal for users and the communities is?
> > >> > >
> > >> > >   - Do we want to help build a relationship between the Pulsar and
> > >> Flink
> > >> > > open source communities?
> > >> > >   - Will users find a connector in the streamnative repository?
> > >> > >   - Will users trust a connector that is not part of Flink as
> much?
> > >> > >
> > >> > > And then decide what is best according to the overall goals there.
> > >> > > As mentioned before, I believe that we can handle connectors more
> > >> > > pragmatically and less strict than the core of Flink, if it helps
> > >> > unlocking
> > >> > > users.
> > >> > >
> > >> > > Best,
> > >> > > Stephan
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]>
> > wrote:
> > >> > >
> > >> > > > Thanks Becket.
> > >> > > >
> > >> > > > I think it is better for the Flink community to judge the
> > benefits of
> > >> > > doing
> > >> > > > this. I was trying to provide some views from outsiders.
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Sijie
> > >> > > >
> > >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <
> [hidden email]
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi Sijie,
> > >> > > > >
> > >> > > > > Yes, we will have to support existing old connectors and new
> > >> > connectors
> > >> > > > in
> > >> > > > > parallel for a while. We have to take that maintenance
> overhead
> > >> > because
> > >> > > > > existing connectors have been used by the users for a long
> > time. I
> > >> > > guess
> > >> > > > It
> > >> > > > > may take at least a year for us to fully remove the old
> > connectors.
> > >> > > > >
> > >> > > > > Process wise, we can do the same for Pulsar connector. But I
> am
> > not
> > >> > > sure
> > >> > > > if
> > >> > > > > we want to have the same burden on Pulsar connector, and I
> would
> > >> like
> > >> > > to
> > >> > > > > understand the benefit of doing that.
> > >> > > > >
> > >> > > > > For users, the benefit of having the old Pulsar connector
> > checked
> > >> in
> > >> > > > seems
> > >> > > > > limited because 1) that code base will be immediately
> > deprecated in
> > >> > the
> > >> > > > > next release in 3-4 months; 2) users can always use it even if
> > it
> > >> is
> > >> > > not
> > >> > > > in
> > >> > > > > the Flink code base. Admittedly it is not as convenient as
> > having
> > >> it
> > >> > in
> > >> > > > > Flink code base, but doesn't seem super either. And after 3-4
> > >> months,
> > >> > > > users
> > >> > > > > can just use the new connector in Flink repo.
> > >> > > > >
> > >> > > > > For Flink developers, the old connector code base is not
> > something
> > >> > that
> > >> > > > we
> > >> > > > > want to evolve later. Instead, these code will be deprecated
> and
> > >> > > > > removed. So why do we want to get a beta version out to
> attract
> > >> > people
> > >> > > to
> > >> > > > > use something we don't want to maintain?
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > >
> > >> > > > > Jiangjie (Becket) Qin
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <
> [hidden email]>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Thanks everyone here. Sorry for jumping into the discussion
> > here.
> > >> > > > > >
> > >> > > > > > I am not very familiar about the deprecation process in
> Flink.
> > >> If I
> > >> > > > > > misunderstood the process, please fix me.
> > >> > > > > >
> > >> > > > > > As far as I understand, FLIP-27 is introducing a new unified
> > API
> > >> > for
> > >> > > > > > connectors. After it introduces the new API
> > >> > > > > > and before moving all the existing connectors from old API
> to
> > new
> > >> > > API,
> > >> > > > > both
> > >> > > > > > old ApI and new API will co-exist
> > >> > > > > > for a while until Flink moves all existing connectors to new
> > API.
> > >> > So
> > >> > > > the
> > >> > > > > > Pulsar connector (using old API) can
> > >> > > > > > follow the deprecation process with other connector using
> old
> > API
> > >> > and
> > >> > > > the
> > >> > > > > > deprecation of old API, no?
> > >> > > > > >
> > >> > > > > > If that's the case, I think contributing the current
> connector
> > >> back
> > >> > > to
> > >> > > > > > Flink rather than maintaining it outside Flink
> > >> > > > > > would provide a bit more benefits. We can deprecate the
> > existing
> > >> > > > > > streamnative/pulsar-flink repo and point the users
> > >> > > > > > to use the connector in Flink repo. So all the review
> > processes
> > >> > will
> > >> > > > > happen
> > >> > > > > > within Flink for both old connector and
> > >> > > > > > new connector. It also reduces the confusions for the users
> as
> > >> the
> > >> > > > > > documentation and code base happen in one place.
> > >> > > > > >
> > >> > > > > > Thoughts?
> > >> > > > > >
> > >> > > > > > - Sijie
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <
> > >> [hidden email]>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Thanks for the explanation, Stephan. I have a few
> questions
> > /
> > >> > > > thoughts.
> > >> > > > > > >
> > >> > > > > > > So that means we will remove the old connector without a
> > major
> > >> > > > version
> > >> > > > > > > bump, is that correct?
> > >> > > > > > >
> > >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11
> > >> connectors
> > >> > > > will
> > >> > > > > > > always work because we saw some dependency class
> collisions
> > in
> > >> > the
> > >> > > > > past.
> > >> > > > > > To
> > >> > > > > > > make it safe we may have to maintain the old code for one
> > more
> > >> > > > release.
> > >> > > > > > >
> > >> > > > > > > To be honest I am still wondering if we have to put the
> old
> > >> > > connector
> > >> > > > > in
> > >> > > > > > > Flink repo. if we check in the old connector to Flink. We
> > will
> > >> > end
> > >> > > up
> > >> > > > > in
> > >> > > > > > > the following situation:
> > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > >> > > > > > > 2. Old connector in Flink Repo, which may be different
> from
> > the
> > >> > one
> > >> > > > in
> > >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed
> in
> > >> 1.12)
> > >> > > > > > > 3. New connector in Flink Repo.
> > >> > > > > > >
> > >> > > > > > > We need to think about how to make the users in each case
> > >> happy.
> > >> > > > > > > - For users of (1), I assume Sijie and Yijie will have to
> > >> > maintain
> > >> > > > the
> > >> > > > > > code
> > >> > > > > > > a bit longer for its own compatibility even after we have
> > (2).
> > >> In
> > >> > > > that
> > >> > > > > > > case, bugs found in old connector may or may not need to
> be
> > >> fixed
> > >> > > in
> > >> > > > > both
> > >> > > > > > > Flink and the streamnative/pulsar-flink repo.
> > >> > > > > > > - For users of (2), will we provide bug fixes? If we do,
> it
> > >> will
> > >> > > be a
> > >> > > > > > > little awkward because those bug fixes will be immediately
> > >> > > deprecated
> > >> > > > > in
> > >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking
> > users
> > >> to
> > >> > > > > migrate
> > >> > > > > > > away from the bug fix. After Flink 1.12, users may still
> > have
> > >> to
> > >> > > > switch
> > >> > > > > > to
> > >> > > > > > > use (3) due to the potential dependency class conflicts
> > >> mentioned
> > >> > > > > above.
> > >> > > > > > > - Users of (3) have a much easier life and don't need to
> > worry
> > >> > too
> > >> > > > > much.
> > >> > > > > > >
> > >> > > > > > > The above story seems a little complicated to tell. I
> think
> > it
> > >> > will
> > >> > > > be
> > >> > > > > > much
> > >> > > > > > > easier to not have (2) at all.
> > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > >> > > > > > > 3. New connector in Flink Repo.
> > >> > > > > > >
> > >> > > > > > > - Old connector will only be maintained in
> > >> > > streamnative/pulsar-flink
> > >> > > > > repo
> > >> > > > > > > until it is fully deprecated. Users can always use the
> > existing
> > >> > > > Pulsar
> > >> > > > > > > connector in that repo.
> > >> > > > > > > - New connector will be in Flink repo and maintained like
> > the
> > >> > other
> > >> > > > > > > connectors.
> > >> > > > > > >
> > >> > > > > > > This seems much simpler for users understand and they will
> > not
> > >> be
> > >> > > > > blocked
> > >> > > > > > > from using the old connector. If the concern is about the
> > >> quality
> > >> > > of
> > >> > > > > the
> > >> > > > > > > connector in streamnative/pulsar-flink repo, is it enough
> > for
> > >> us
> > >> > > just
> > >> > > > > to
> > >> > > > > > > review the code in streamnative/pulsar-flink connector to
> > make
> > >> > sure
> > >> > > > it
> > >> > > > > > > looks good from Flink's perspective?
> > >> > > > > > >
> > >> > > > > > > What do you think?
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > >
> > >> > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <
> > [hidden email]
> > >> >
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > My take would be the following:
> > >> > > > > > > >
> > >> > > > > > > >   - If we merge the connector now and replace it with a
> > >> FLIP-27
> > >> > > > > version
> > >> > > > > > > > before the 1.10 release, then we need no deprecation
> > process
> > >> > > > > > > >   - If we don't manage to replace it with a FLIP-27
> > version
> > >> > > before
> > >> > > > > the
> > >> > > > > > > 1.10
> > >> > > > > > > > release, than it is good that we have the other version,
> > so
> > >> no
> > >> > > > users
> > >> > > > > > get
> > >> > > > > > > > blocked.
> > >> > > > > > > >
> > >> > > > > > > > In the latter case we can see how we want to do it.
> > Immediate
> > >> > > > removal
> > >> > > > > > of
> > >> > > > > > > > the old version or deprecation label and keeping it for
> > one
> > >> > more
> > >> > > > > > release.
> > >> > > > > > > > Given that you should be able to use a Flink 1.10
> > connector
> > >> > with
> > >> > > > > Flink
> > >> > > > > > > 1.11
> > >> > > > > > > > as well (stable public APIs) there is also a workaround
> if
> > >> you
> > >> > > need
> > >> > > > > an
> > >> > > > > > > old
> > >> > > > > > > > connector in a newer version. So immediate removal might
> > even
> > >> > be
> > >> > > > > > > feasible.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
> > >> > > [hidden email]>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Stephan,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the clarification. I completely agree with
> > you
> > >> and
> > >> > > > > Thomas
> > >> > > > > > on
> > >> > > > > > > > the
> > >> > > > > > > > > process of adding connectors to Flink repo. However, I
> > am
> > >> > > > wondering
> > >> > > > > > > what
> > >> > > > > > > > is
> > >> > > > > > > > > the deprecation process? Given the main concern here
> was
> > >> that
> > >> > > we
> > >> > > > > may
> > >> > > > > > > have
> > >> > > > > > > > > to maintain two Pulsar connector code bases until the
> > old
> > >> one
> > >> > > is
> > >> > > > > > > removed
> > >> > > > > > > > > from the repo, it would be good to know how long we
> > have to
> > >> > do
> > >> > > > > that.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
> > >> > [hidden email]
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Some quick thoughts on the connector contribution
> > >> process.
> > >> > I
> > >> > > > > > > basically
> > >> > > > > > > > > > reiterate here what Thomas mentioned in another
> thread
> > >> > about
> > >> > > > the
> > >> > > > > > > > Kinesis
> > >> > > > > > > > > > connector.
> > >> > > > > > > > > >
> > >> > > > > > > > > > For connectors, we should favor a low-overhead
> > >> contribution
> > >> > > > > > process,
> > >> > > > > > > > and
> > >> > > > > > > > > > accept user code and changes more readily than in
> the
> > >> core
> > >> > > > > system.
> > >> > > > > > > > > > That is because connectors have both a big variety
> of
> > >> > > scenarios
> > >> > > > > > they
> > >> > > > > > > > get
> > >> > > > > > > > > > used in (only through use and many small
> > contributions do
> > >> > > they
> > >> > > > > > become
> > >> > > > > > > > > > really useful over time) and at the same time, and
> > >> > committers
> > >> > > > do
> > >> > > > > > not
> > >> > > > > > > > use
> > >> > > > > > > > > > the connector themselves and usually cannot foresee
> > too
> > >> > well
> > >> > > > what
> > >> > > > > > is
> > >> > > > > > > > > > needed.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Further more, a missing connector (or connector
> > feature)
> > >> is
> > >> > > > > often a
> > >> > > > > > > > > bigger
> > >> > > > > > > > > > show stopper for users than a missing API or system
> > >> > feature.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Along these lines of thougt, the conclusion would be
> > to
> > >> > take
> > >> > > > the
> > >> > > > > > > Pulsar
> > >> > > > > > > > > > connector now, focus the review on
> > >> legal/dependencies/rough
> > >> > > > code
> > >> > > > > > > style
> > >> > > > > > > > > and
> > >> > > > > > > > > > conventions, label it as "beta" (in the sense of
> "new
> > >> code"
> > >> > > > that
> > >> > > > > is
> > >> > > > > > > > "not
> > >> > > > > > > > > > yet tested through longer use") and go ahead. And
> then
> > >> > evolve
> > >> > > > it
> > >> > > > > > > > quickly
> > >> > > > > > > > > > without putting formal blockers in the way, meaning
> > also
> > >> > > > adding a
> > >> > > > > > new
> > >> > > > > > > > > FLIP
> > >> > > > > > > > > > 27 version when it is there.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Best,
> > >> > > > > > > > > > Stephan
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> > >> > > > [hidden email]
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Yijie,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Could you please follow the FLIP process to start
> a
> > new
> > >> > > FLIP
> > >> > > > > > > > > [DISCUSSION]
> > >> > > > > > > > > > > thread in the mailing list?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list
> > now.
> > >> So
> > >> > > > there
> > >> > > > > > is a
> > >> > > > > > > > > FLIP
> > >> > > > > > > > > > > number collision. Can you change the FLIP number
> to
> > 72?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> > >> > > > > [hidden email]>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Hi Yijie,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP.
> > >> > > > > > > > > > > > Would you mind enabling comments/suggestions on
> > the
> > >> > > google
> > >> > > > > doc
> > >> > > > > > > > link?
> > >> > > > > > > > > > This
> > >> > > > > > > > > > > > way the contributors from the community can
> > comment
> > >> on
> > >> > > the
> > >> > > > > doc.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > Rong
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > >> > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hello everyone,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I've drafted a FLIP that describes the current
> > >> design
> > >> > > of
> > >> > > > > the
> > >> > > > > > > > Pulsar
> > >> > > > > > > > > > > > > connector:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Please take a look and let me know what you
> > think.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > Yijie
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > >> > > > > > > [hidden email]>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi All,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Sorry for joining the discussion late and
> > thanks
> > >> > > Yijie
> > >> > > > &
> > >> > > > > > > Sijie
> > >> > > > > > > > > for
> > >> > > > > > > > > > > > > driving
> > >> > > > > > > > > > > > > > the discussion.
> > >> > > > > > > > > > > > > > I also think the Pulsar connector would be a
> > very
> > >> > > > > valuable
> > >> > > > > > > > > addition
> > >> > > > > > > > > > > to
> > >> > > > > > > > > > > > > > Flink. I can also help out a bit on the
> review
> > >> side
> > >> > > :-)
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Regarding the timeline, I also share
> concerns
> > >> with
> > >> > > > Becket
> > >> > > > > > on
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > > relationship between the new Pulsar
> connector
> > and
> > >> > > > > FLIP-27.
> > >> > > > > > > > > > > > > > There's also another discussion just started
> > by
> > >> > > Stephan
> > >> > > > > on
> > >> > > > > > > > > dropping
> > >> > > > > > > > > > > > Kafka
> > >> > > > > > > > > > > > > > 9/10 support on next Flink release [1].
> > Although
> > >> > the
> > >> > > > > > > situation
> > >> > > > > > > > > is
> > >> > > > > > > > > > > > > somewhat
> > >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has been
> > in
> > >> > Flink
> > >> > > > for
> > >> > > > > > > > almost
> > >> > > > > > > > > > 3-4
> > >> > > > > > > > > > > > > years,
> > >> > > > > > > > > > > > > > based on the discussion I am not sure if a
> > major
> > >> > > > version
> > >> > > > > > > > release
> > >> > > > > > > > > > is a
> > >> > > > > > > > > > > > > > requirement for removing old connector
> > supports.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I think there shouldn't be a blocker if we
> > agree
> > >> > the
> > >> > > > old
> > >> > > > > > > > > connector
> > >> > > > > > > > > > > will
> > >> > > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector
> is
> > >> > there.
> > >> > > > As
> > >> > > > > > > > Stephan
> > >> > > > > > > > > > > > stated,
> > >> > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > is easier to contribute the source sooner
> and
> > >> > adjust
> > >> > > it
> > >> > > > > > > later.
> > >> > > > > > > > > > > > > > We should also ensure we clearly communicate
> > the
> > >> > > > message:
> > >> > > > > > for
> > >> > > > > > > > > > > example,
> > >> > > > > > > > > > > > > > putting an experimental flag on the
> pre-FLIP27
> > >> > > > connector
> > >> > > > > > page
> > >> > > > > > > > of
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > > > website, documentations, etc. Any other
> > thoughts?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > --
> > >> > > > > > > > > > > > > > Rong
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > [1]
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin <
> > >> > > > > > > > [hidden email]
> > >> > > > > > > > > >
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Technically speaking, removing the old
> > >> connector
> > >> > > code
> > >> > > > > is
> > >> > > > > > a
> > >> > > > > > > > > > > backwards
> > >> > > > > > > > > > > > > > > incompatible change which requires a major
> > >> > version
> > >> > > > > bump,
> > >> > > > > > > i.e.
> > >> > > > > > > > > > Flink
> > >> > > > > > > > > > > > > 2.x.
> > >> > > > > > > > > > > > > > > Given that we don't have a clear plan on
> > when
> > >> to
> > >> > > have
> > >> > > > > the
> > >> > > > > > > > next
> > >> > > > > > > > > > > major
> > >> > > > > > > > > > > > > > > version release, it seems unclear how long
> > the
> > >> > old
> > >> > > > > > > connector
> > >> > > > > > > > > code
> > >> > > > > > > > > > > > will
> > >> > > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > > there if we check it in right now. Or will
> > we
> > >> > > remove
> > >> > > > > the
> > >> > > > > > > old
> > >> > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > without a major version bump? In any case,
> > it
> > >> > > sounds
> > >> > > > > not
> > >> > > > > > > > quite
> > >> > > > > > > > > > user
> > >> > > > > > > > > > > > > > > friendly to the those who might use the
> old
> > >> > Pulsar
> > >> > > > > > > > connector. I
> > >> > > > > > > > > > am
> > >> > > > > > > > > > > > not
> > >> > > > > > > > > > > > > sure
> > >> > > > > > > > > > > > > > > if it is worth these potential problems in
> > >> order
> > >> > to
> > >> > > > > have
> > >> > > > > > > the
> > >> > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > source
> > >> > > > > > > > > > > > > > > connector checked in one or two months
> > earlier.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan
> > Ewen <
> > >> > > > > > > > [hidden email]
> > >> > > > > > > > > >
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > Agreed, if we check in the old code, we
> > >> should
> > >> > > make
> > >> > > > > it
> > >> > > > > > > > clear
> > >> > > > > > > > > > that
> > >> > > > > > > > > > > > it
> > >> > > > > > > > > > > > > will
> > >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based
> > >> version
> > >> > > of
> > >> > > > > the
> > >> > > > > > > > > > connector
> > >> > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > there.
> > >> > > > > > > > > > > > > > > > We should not commit to maintaining the
> > old
> > >> > > > version,
> > >> > > > > > that
> > >> > > > > > > > > would
> > >> > > > > > > > > > > be
> > >> > > > > > > > > > > > > indeed
> > >> > > > > > > > > > > > > > > > too much overhead.
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket
> > Qin <
> > >> > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Hi Stephan,
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Thanks for the volunteering to help.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Yes, the overhead would just be review
> > >> > > capacity.
> > >> > > > In
> > >> > > > > > > > fact, I
> > >> > > > > > > > > > am
> > >> > > > > > > > > > > > not
> > >> > > > > > > > > > > > > > > > worrying
> > >> > > > > > > > > > > > > > > > > too much about the review capacity.
> > That is
> > >> > > just
> > >> > > > a
> > >> > > > > > one
> > >> > > > > > > > time
> > >> > > > > > > > > > > cost.
> > >> > > > > > > > > > > > > My
> > >> > > > > > > > > > > > > > > > > concern is mainly about the long term
> > >> burden.
> > >> > > > > Assume
> > >> > > > > > we
> > >> > > > > > > > > have
> > >> > > > > > > > > > > new
> > >> > > > > > > > > > > > > source
> > >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly
> added
> > >> > Pulsar
> > >> > > > > > > > connectors
> > >> > > > > > > > > in
> > >> > > > > > > > > > > old
> > >> > > > > > > > > > > > > > > > > interface. Later on if we migrate
> > Pulsar to
> > >> > new
> > >> > > > > > source
> > >> > > > > > > > > > > interface,
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > old
> > >> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated
> > almost
> > >> > > > > > immediately
> > >> > > > > > > > > after
> > >> > > > > > > > > > > > > checked
> > >> > > > > > > > > > > > > > > in,
> > >> > > > > > > > > > > > > > > > > but we may still have to maintain two
> > code
> > >> > > bases.
> > >> > > > > For
> > >> > > > > > > the
> > >> > > > > > > > > > > > existing
> > >> > > > > > > > > > > > > > > > > connectors, we have to do that
> anyways.
> > But
> > >> > it
> > >> > > > > would
> > >> > > > > > be
> > >> > > > > > > > > good
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > > avoid
> > >> > > > > > > > > > > > > > > > > introducing a new connector with the
> > same
> > >> > > > problem.
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM
> Stephan
> > >> Ewen
> > >> > <
> > >> > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Hi all!
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Nice to see this lively discussion
> > about
> > >> > the
> > >> > > > > Pulsar
> > >> > > > > > > > > > > connector.
> > >> > > > > > > > > > > > > > > > > > Some thoughts on the open questions:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain
> as
> > a
> > >> > > > community
> > >> > > > > > > > package
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Looks like the discussion is more
> > going
> > >> > > towards
> > >> > > > > > > > > > > contribution. I
> > >> > > > > > > > > > > > > think
> > >> > > > > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > > > > is good, especially if we think that
> > we
> > >> > want
> > >> > > to
> > >> > > > > > > build a
> > >> > > > > > > > > > > > similarly
> > >> > > > > > > > > > > > > > > deep
> > >> > > > > > > > > > > > > > > > > > integration with Pulsar as we have
> for
> > >> > > example
> > >> > > > > with
> > >> > > > > > > > > Kafka.
> > >> > > > > > > > > > > The
> > >> > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > already looks like a more thorough
> > >> > connector
> > >> > > > than
> > >> > > > > > > many
> > >> > > > > > > > > > others
> > >> > > > > > > > > > > > we
> > >> > > > > > > > > > > > > have
> > >> > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > the repository.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > With either a repo split, or the new
> > >> build
> > >> > > > > system,
> > >> > > > > > I
> > >> > > > > > > > hope
> > >> > > > > > > > > > > that
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > build
> > >> > > > > > > > > > > > > > > > > > overhead is not a problem.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > ## Committer Support
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Becket offered some help already, I
> > can
> > >> > also
> > >> > > > > help a
> > >> > > > > > > > bit.
> > >> > > > > > > > > I
> > >> > > > > > > > > > > hope
> > >> > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > > > > between us, we can cover this.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for
> FLIP-27
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually
> > >> making
> > >> > > some
> > >> > > > > > > PoC-ing
> > >> > > > > > > > > > > > > progress, but
> > >> > > > > > > > > > > > > > > > > will
> > >> > > > > > > > > > > > > > > > > > take 2 more months, I would
> estimate,
> > >> > before
> > >> > > it
> > >> > > > > is
> > >> > > > > > > > fully
> > >> > > > > > > > > > > > > available.
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > If we want to be on the safe side
> with
> > >> the
> > >> > > > > > > > contribution,
> > >> > > > > > > > > we
> > >> > > > > > > > > > > > > should
> > >> > > > > > > > > > > > > > > > > > contribute the source sooner and
> > adjust
> > >> it
> > >> > > > later.
> > >> > > > > > > That
> > >> > > > > > > > > > would
> > >> > > > > > > > > > > > also
> > >> > > > > > > > > > > > > > > help
> > >> > > > > > > > > > > > > > > > us
> > >> > > > > > > > > > > > > > > > > > in case things get crazy towards the
> > 1.10
> > >> > > > feature
> > >> > > > > > > > freeze
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > it
> > >> > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > > > > > hard to find time to review the new
> > >> > changes.
> > >> > > > > > > > > > > > > > > > > > What would be the overhead of
> > >> contributing
> > >> > > now?
> > >> > > > > > Given
> > >> > > > > > > > > that
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > code
> > >> > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > already there, it looks like it
> would
> > be
> > >> > only
> > >> > > > > > review
> > >> > > > > > > > > > > capacity,
> > >> > > > > > > > > > > > > right?
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > > > > Stephan
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM
> Yijie
> > >> > Shen <
> > >> > > > > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > Hi everyone!
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > Thanks for your attention and the
> > >> > promotion
> > >> > > > of
> > >> > > > > > this
> > >> > > > > > > > > work.
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as
> > >> > possible
> > >> > > > for
> > >> > > > > > more
> > >> > > > > > > > > > > specific
> > >> > > > > > > > > > > > > > > > > discussions.
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have
> > not
> > >> > > > reached
> > >> > > > > a
> > >> > > > > > > > > > consensus.
> > >> > > > > > > > > > > > > > > > Therefore,
> > >> > > > > > > > > > > > > > > > > > > I will explain all the
> > functionalities
> > >> of
> > >> > > the
> > >> > > > > > > > existing
> > >> > > > > > > > > > > > > connector in
> > >> > > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink,
> > and
> > >> > > > Catalog)
> > >> > > > > to
> > >> > > > > > > > > > continue
> > >> > > > > > > > > > > > our
> > >> > > > > > > > > > > > > > > > > > > discussions in FLIP.
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > Thanks for your kind help.
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > > > > > Yijie
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM
> > Becket
> > >> > Qin
> > >> > > <
> > >> > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > Hi Sijie,
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is to
> > have
> > >> > > Pulsar
> > >> > > > > > > > connector
> > >> > > > > > > > > > in
> > >> > > > > > > > > > > > > 1.10,
> > >> > > > > > > > > > > > > > > how
> > >> > > > > > > > > > > > > > > > > > about
> > >> > > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > do the following:
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar
> > >> connector
> > >> > > to
> > >> > > > > > Flink
> > >> > > > > > > > main
> > >> > > > > > > > > > > repo
> > >> > > > > > > > > > > > > as it
> > >> > > > > > > > > > > > > > > > is a
> > >> > > > > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > > public interface to Flink main
> > repo.
> > >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar
> sink
> > >> > right
> > >> > > > away
> > >> > > > > > as
> > >> > > > > > > > > there
> > >> > > > > > > > > > is
> > >> > > > > > > > > > > > no
> > >> > > > > > > > > > > > > > > change
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > sink interface so far.
> > >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27.
> > >> Flink
> > >> > > 1.10
> > >> > > > > is
> > >> > > > > > > > going
> > >> > > > > > > > > to
> > >> > > > > > > > > > > be
> > >> > > > > > > > > > > > > code
> > >> > > > > > > > > > > > > > > > > freeze
> > >> > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we give a
> > >> month
> > >> > to
> > >> > > > the
> > >> > > > > > > > > > development
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > review
> > >> > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to
> have
> > >> > FLIP-27
> > >> > > > by
> > >> > > > > > late
> > >> > > > > > > > > Oct.
> > >> > > > > > > > > > > > There
> > >> > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > still
> > >> > > > > > > > > > > > > > > > > > 7
> > >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it is
> > >> doable.
> > >> > > If
> > >> > > > > > > FLIP-27
> > >> > > > > > > > is
> > >> > > > > > > > > > not
> > >> > > > > > > > > > > > > ready
> > >> > > > > > > > > > > > > > > by
> > >> > > > > > > > > > > > > > > > > late
> > >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check in
> > >> Pulsar
> > >> > > > > > connector
> > >> > > > > > > > with
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > > > > existing
> > >> > > > > > > > > > > > > > > > > > source
> > >> > > > > > > > > > > > > > > > > > > > interface. This means we will
> have
> > >> > Pulsar
> > >> > > > > > > connector
> > >> > > > > > > > > in
> > >> > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > 1.10,
> > >> > > > > > > > > > > > > > > > > > either
> > >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > Because we are going to have
> > Pulsar
> > >> > sink
> > >> > > > and
> > >> > > > > > > source
> > >> > > > > > > > > > > checked
> > >> > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > separately,
> > >> > > > > > > > > > > > > > > > > > > > it might make sense to have two
> > >> FLIPs,
> > >> > > one
> > >> > > > > for
> > >> > > > > > > > Pulsar
> > >> > > > > > > > > > > sink
> > >> > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > another
> > >> > > > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can start
> > the
> > >> > work
> > >> > > on
> > >> > > > > > > Pulsar
> > >> > > > > > > > > sink
> > >> > > > > > > > > > > > right
> > >> > > > > > > > > > > > > > > away.
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM
> > Sijie
> > >> > Guo
> > >> > > <
> > >> > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink
> > >> community?
> > >> > > > Shall
> > >> > > > > > we
> > >> > > > > > > > wait
> > >> > > > > > > > > > for
> > >> > > > > > > > > > > > > FLIP-27
> > >> > > > > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > > > > > > shall we
> > >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And
> what
> > the
> > >> > > next
> > >> > > > > > steps
> > >> > > > > > > > are?
> > >> > > > > > > > > > :-)
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > > > > > > Sijie
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM
> > >> Bowen
> > >> > > Li <
> > >> > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > Hi,
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar
> > connector
> > >> > in
> > >> > > > > Flink
> > >> > > > > > > can
> > >> > > > > > > > > be a
> > >> > > > > > > > > > > > good
> > >> > > > > > > > > > > > > > > mutual
> > >> > > > > > > > > > > > > > > > > > > benefit
> > >> > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > both communities.
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > Another perspective is that
> > >> Pulsar
> > >> > > > > > connector
> > >> > > > > > > is
> > >> > > > > > > > > the
> > >> > > > > > > > > > > 1st
> > >> > > > > > > > > > > > > > > > streaming
> > >> > > > > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > > > > that integrates with Flink's
> > >> > metadata
> > >> > > > > > > > management
> > >> > > > > > > > > > > system
> > >> > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > Catalog
> > >> > > > > > > > > > > > > > > > > > > APIs.
> > >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the
> > >> > > > integration
> > >> > > > > > > turns
> > >> > > > > > > > > out
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > whether
> > >> > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > need to
> > >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack,
> > >> which
> > >> > > are
> > >> > > > > > > > currently
> > >> > > > > > > > > in
> > >> > > > > > > > > > > > > Beta, to
> > >> > > > > > > > > > > > > > > > > cater
> > >> > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus
> > I'm
> > >> in
> > >> > > > favor
> > >> > > > > of
> > >> > > > > > > > > merging
> > >> > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > into
> > >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10.
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit
> smaller
> > >> sized
> > >> > > > PRs,
> > >> > > > > > e.g.
> > >> > > > > > > > > maybe
> > >> > > > > > > > > > > one
> > >> > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > basic
> > >> > > > > > > > > > > > > > > > > > > > > > source/sink functionalities
> > and
> > >> > > another
> > >> > > > > for
> > >> > > > > > > > > schema
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > catalog
> > >> > > > > > > > > > > > > > > > > > > > > integration,
> > >> > > > > > > > > > > > > > > > > > > > > > just to make them easier to
> > >> review.
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to
> > wait
> > >> for
> > >> > > > > > FLIP-27.
> > >> > > > > > > > But
> > >> > > > > > > > > I
> > >> > > > > > > > > > > > don't
> > >> > > > > > > > > > > > > > > think
> > >> > > > > > > > > > > > > > > > > > > FLIP-27
> > >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in cases
> > >> where
> > >> > it
> > >> > > > > > cannot
> > >> > > > > > > > make
> > >> > > > > > > > > > its
> > >> > > > > > > > > > > > way
> > >> > > > > > > > > > > > > > > into
> > >> > > > > > > > > > > > > > > > > 1.10
> > >> > > > > > > > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable
> > amount
> > >> of
> > >> > > time
> > >> > > > > for
> > >> > > > > > > > > > > committers
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > review
> > >> > > > > > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully
> > adapt
> > >> to
> > >> > > new
> > >> > > > > > > > > interfaces.
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > Bowen
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21
> AM
> > >> > Becket
> > >> > > > > Qin <
> > >> > > > > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > Hi Till,
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > You are right. It all
> > depends
> > >> on
> > >> > > when
> > >> > > > > the
> > >> > > > > > > new
> > >> > > > > > > > > > > source
> > >> > > > > > > > > > > > > > > > interface
> > >> > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > going
> > >> > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I
> > think it
> > >> > > would
> > >> > > > > be
> > >> > > > > > > > there
> > >> > > > > > > > > in
> > >> > > > > > > > > > > > > about a
> > >> > > > > > > > > > > > > > > > month
> > >> > > > > > > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > > > > > > so.
> > >> > > > > > > > > > > > > > > > > > > > > > But
> > >> > > > > > > > > > > > > > > > > > > > > > > I could be too optimistic.
> > It
> > >> > would
> > >> > > > > also
> > >> > > > > > be
> > >> > > > > > > > > good
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > > hear
> > >> > > > > > > > > > > > > > > what
> > >> > > > > > > > > > > > > > > > > do
> > >> > > > > > > > > > > > > > > > > > > > > Aljoscha
> > >> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as they
> > are
> > >> > also
> > >> > > > > > involved
> > >> > > > > > > > in
> > >> > > > > > > > > > > > FLIP-27.
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > In general I think we
> should
> > >> have
> > >> > > > > Pulsar
> > >> > > > > > > > > > connector
> > >> > > > > > > > > > > in
> > >> > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > 1.10,
> > >> > > > > > > > > > > > > > > > > > > > > > > preferably with the new
> > source
> > >> > > > > interface.
> > >> > > > > > > We
> > >> > > > > > > > > can
> > >> > > > > > > > > > > also
> > >> > > > > > > > > > > > > check
> > >> > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > right
> > >> > > > > > > > > > > > > > > > > > > > > > now
> > >> > > > > > > > > > > > > > > > > > > > > > > with old source interface,
> > but
> > >> I
> > >> > > > > suspect
> > >> > > > > > > few
> > >> > > > > > > > > > users
> > >> > > > > > > > > > > > > will use
> > >> > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > before
> > >> > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > next official release.
> > >> Therefore,
> > >> > > it
> > >> > > > > > seems
> > >> > > > > > > > > > > reasonable
> > >> > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > wait a
> > >> > > > > > > > > > > > > > > > > > > little
> > >> > > > > > > > > > > > > > > > > > > > > > bit
> > >> > > > > > > > > > > > > > > > > > > > > > > to see whether we can jump
> > to
> > >> the
> > >> > > new
> > >> > > > > > > source
> > >> > > > > > > > > > > > > interface. As
> > >> > > > > > > > > > > > > > > > long
> > >> > > > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > > > make
> > >> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it,
> > >> waiting a
> > >> > > > > little
> > >> > > > > > > bit
> > >> > > > > > > > > > > doesn't
> > >> > > > > > > > > > > > > seem
> > >> > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > hurt
> > >> > > > > > > > > > > > > > > > > > > much.
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> 3:59
> > PM
> > >> > Till
> > >> > > > > > > Rohrmann
> > >> > > > > > > > <
> > >> > > > > > > > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the
> > >> problem
> > >> > > > would
> > >> > > > > be
> > >> > > > > > > if
> > >> > > > > > > > we
> > >> > > > > > > > > > > > > committed
> > >> > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > connector before the new
> > >> source
> > >> > > > > > interface
> > >> > > > > > > > is
> > >> > > > > > > > > > > ready.
> > >> > > > > > > > > > > > > If I
> > >> > > > > > > > > > > > > > > > > > > understood
> > >> > > > > > > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we need
> to
> > >> > > support
> > >> > > > > the
> > >> > > > > > > old
> > >> > > > > > > > > > source
> > >> > > > > > > > > > > > > > > interface
> > >> > > > > > > > > > > > > > > > > > > anyway
> > >> > > > > > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > existing connectors. By
> > >> > checking
> > >> > > it
> > >> > > > > in
> > >> > > > > > > > early
> > >> > > > > > > > > I
> > >> > > > > > > > > > > > could
> > >> > > > > > > > > > > > > see
> > >> > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > benefit
> > >> > > > > > > > > > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > > > > > > > > > > our users could start
> > using
> > >> the
> > >> > > > > > connector
> > >> > > > > > > > > > > earlier.
> > >> > > > > > > > > > > > > > > > Moreover,
> > >> > > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar
> > >> > > integration
> > >> > > > > is
> > >> > > > > > > > being
> > >> > > > > > > > > > > > delayed
> > >> > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > case
> > >> > > > > > > > > > > > > > > > > > > that the
> > >> > > > > > > > > > > > > > > > > > > > > > > > source interface should
> be
> > >> > > delayed.
> > >> > > > > The
> > >> > > > > > > > only
> > >> > > > > > > > > > > > > downside I
> > >> > > > > > > > > > > > > > > see
> > >> > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > extra
> > >> > > > > > > > > > > > > > > > > > > > > > > > review effort and
> > potential
> > >> > fixes
> > >> > > > > which
> > >> > > > > > > > might
> > >> > > > > > > > > > be
> > >> > > > > > > > > > > > > > > irrelevant
> > >> > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > > > > > > source interface
> > >> > implementation.
> > >> > > I
> > >> > > > > > guess
> > >> > > > > > > it
> > >> > > > > > > > > > > mainly
> > >> > > > > > > > > > > > > > > depends
> > >> > > > > > > > > > > > > > > > on
> > >> > > > > > > > > > > > > > > > > > how
> > >> > > > > > > > > > > > > > > > > > > > > > certain
> > >> > > > > > > > > > > > > > > > > > > > > > > > we are when the new
> source
> > >> > > > interface
> > >> > > > > > will
> > >> > > > > > > > be
> > >> > > > > > > > > > > ready.
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > > > > > > > > > > > > Till
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> > 8:56
> > >> AM
> > >> > > > Becket
> > >> > > > > > > Qin <
> > >> > > > > > > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing
> your
> > >> > > thoughts.
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some
> > >> update
> > >> > > on
> > >> > > > > > > FLIP-27.
> > >> > > > > > > > > > > > Although
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > FLIP
> > >> > > > > > > > > > > > > > > > > > > wiki
> > >> > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread has
> > been
> > >> > > quiet
> > >> > > > > for
> > >> > > > > > > some
> > >> > > > > > > > > > > time, a
> > >> > > > > > > > > > > > > few
> > >> > > > > > > > > > > > > > > > > > > committer /
> > >> > > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink
> > >> > community
> > >> > > > > were
> > >> > > > > > > > > actually
> > >> > > > > > > > > > > > > > > prototyping
> > >> > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > entire
> > >> > > > > > > > > > > > > > > > > > > > > > > > thing.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > We have made some good
> > >> > progress
> > >> > > > > there
> > >> > > > > > > but
> > >> > > > > > > > > > want
> > >> > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > update
> > >> > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > FLIP
> > >> > > > > > > > > > > > > > > > > > > > > > wiki
> > >> > > > > > > > > > > > > > > > > > > > > > > > > after the entire thing
> > is
> > >> > > > verified
> > >> > > > > to
> > >> > > > > > > > work
> > >> > > > > > > > > in
> > >> > > > > > > > > > > > case
> > >> > > > > > > > > > > > > > > there
> > >> > > > > > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > > some
> > >> > > > > > > > > > > > > > > > > > > > > > last
> > >> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the
> > >> > > > > > implementation.
> > >> > > > > > > I
> > >> > > > > > > > > > don't
> > >> > > > > > > > > > > > > have an
> > >> > > > > > > > > > > > > > > > > exact
> > >> > > > > > > > > > > > > > > > > > > ETA
> > >> > > > > > > > > > > > > > > > > > > > > yet,
> > >> > > > > > > > > > > > > > > > > > > > > > > > but I
> > >> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to
> be
> > >> > within
> > >> > > a
> > >> > > > > > month
> > >> > > > > > > or
> > >> > > > > > > > > so.
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review
> the
> > >> > > current
> > >> > > > > > Flink
> > >> > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > see if
> > >> > > > > > > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It
> > would be
> > >> > > good
> > >> > > > to
> > >> > > > > > > avoid
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > case
> > >> > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > checked
> > >> > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with
> > some
> > >> > > review
> > >> > > > > > > efforts
> > >> > > > > > > > > and
> > >> > > > > > > > > > > > > shortly
> > >> > > > > > > > > > > > > > > > after
> > >> > > > > > > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is
> > ready.
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> > 8:39
> > >> > AM
> > >> > > > > Yijie
> > >> > > > > > > > Shen <
> > >> > > > > > > > > > > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the
> > >> feedback
> > >> > > and
> > >> > > > > > > > > > suggestions!
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the
> > goal
> > >> of
> > >> > > the
> > >> > > > > > > > connector
> > >> > > > > > > > > > has
> > >> > > > > > > > > > > > > always
> > >> > > > > > > > > > > > > > > > been
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > provide
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > users with the
> latest
> > >> > > features
> > >> > > > of
> > >> > > > > > > both
> > >> > > > > > > > > > > systems
> > >> > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > soon
> > >> > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > > > possible.
> > >> > > > > > > > > > > > > > > > > > > > > > > We
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > propose to
> contribute
> > the
> > >> > > > > connector
> > >> > > > > > > to
> > >> > > > > > > > > > Flink
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > hope
> > >> > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > get
> > >> > > > > > > > > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and
> > feedback
> > >> > from
> > >> > > > > Flink
> > >> > > > > > > > > experts
> > >> > > > > > > > > > > to
> > >> > > > > > > > > > > > > ensure
> > >> > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > high
> > >> > > > > > > > > > > > > > > > > > > > > > > quality
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we
> > noticed
> > >> its
> > >> > > > > > existence
> > >> > > > > > > > at
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > > > > beginning
> > >> > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > reworking
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > the connector
> > >> > implementation
> > >> > > > > based
> > >> > > > > > on
> > >> > > > > > > > > Flink
> > >> > > > > > > > > > > > 1.9;
> > >> > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > also
> > >> > > > > > > > > > > > > > > > > > > wanted
> > >> > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector
> that
> > >> > > supports
> > >> > > > > > both
> > >> > > > > > > > > batch
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > > stream
> > >> > > > > > > > > > > > > > > > > > > computing
> > >> > > > > > > > > > > > > > > > > > > > > > based
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > on it.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has been
> > >> > inactive
> > >> > > > for
> > >> > > > > > > some
> > >> > > > > > > > > > time,
> > >> > > > > > > > > > > so
> > >> > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > decided
> > >> > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > provide
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with
> most
> > of
> > >> > the
> > >> > > > new
> > >> > > > > > > > > features,
> > >> > > > > > > > > > > such
> > >> > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > type
> > >> > > > > > > > > > > > > > > > > > > > > > > system
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog
> > API
> > >> > > first.
> > >> > > > We
> > >> > > > > > > will
> > >> > > > > > > > > pay
> > >> > > > > > > > > > > > > attention
> > >> > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > progress
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27
> continually
> > >> and
> > >> > > > > > > incorporate
> > >> > > > > > > > it
> > >> > > > > > > > > > > with
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > > > > soon
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > as possible.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test
> > status
> > >> > of
> > >> > > > the
> > >> > > > > > > > > connector,
> > >> > > > > > > > > > > we
> > >> > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > following
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > other
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in
> > Flink
> > >> > > > > > repository
> > >> > > > > > > > and
> > >> > > > > > > > > > > aimed
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > provide
> > >> > > > > > > > > > > > > > > > > > > > > > throughout
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could.
> We
> > are
> > >> > > also
> > >> > > > > > happy
> > >> > > > > > > to
> > >> > > > > > > > > > hear
> > >> > > > > > > > > > > > > > > > suggestions
> > >> > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from the
> > >> Flink
> > >> > > > > > community
> > >> > > > > > > to
> > >> > > > > > > > > > > improve
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > stability and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the
> > >> > connector
> > >> > > > > > > > > continuously.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019
> at
> > >> 5:59
> > >> > > AM
> > >> > > > > > Sijie
> > >> > > > > > > > Guo
> > >> > > > > > > > > <
> > >> > > > > > > > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone
> for
> > the
> > >> > > > > comments
> > >> > > > > > > and
> > >> > > > > > > > > > > > feedback.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me
> that
> > the
> > >> > > main
> > >> > > > > > > question
> > >> > > > > > > > > > here
> > >> > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > about -
> > >> > > > > > > > > > > > > > > > > > "how
> > >> > > > > > > > > > > > > > > > > > > can
> > >> > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community maintain
> > the
> > >> > > > > > connector?".
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two
> > thoughts
> > >> > from
> > >> > > > > > myself.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and
> > >> where
> > >> > to
> > >> > > > > host
> > >> > > > > > > this
> > >> > > > > > > > > > > > > integration
> > >> > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > kind
> > >> > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > less
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > important
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe
> > there
> > >> can
> > >> > > be
> > >> > > > > many
> > >> > > > > > > > ways
> > >> > > > > > > > > to
> > >> > > > > > > > > > > > > achieve
> > >> > > > > > > > > > > > > > > it.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the
> > >> > > contribution,
> > >> > > > > what
> > >> > > > > > > we
> > >> > > > > > > > > are
> > >> > > > > > > > > > > > > looking
> > >> > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > here
> > >> > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > how
> > >> > > > > > > > > > > > > > > > > > > > > > > > these
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > two
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can
> > build
> > >> the
> > >> > > > > > > > collaboration
> > >> > > > > > > > > > > > > > > relationship
> > >> > > > > > > > > > > > > > > > on
> > >> > > > > > > > > > > > > > > > > > > > > > developing
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration
> > between
> > >> > > > Pulsar
> > >> > > > > > and
> > >> > > > > > > > > Flink.
> > >> > > > > > > > > > > > Even
> > >> > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > can
> > >> > > > > > > > > > > > > > > > > try
> > >> > > > > > > > > > > > > > > > > > > our
> > >> > > > > > > > > > > > > > > > > > > > > > best
> > >> > > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > catch
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates
> > in
> > >> > Flink
> > >> > > > > > > > community.
> > >> > > > > > > > > We
> > >> > > > > > > > > > > are
> > >> > > > > > > > > > > > > still
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact
> > that we
> > >> > > have
> > >> > > > > less
> > >> > > > > > > > > > > experiences
> > >> > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > than
> > >> > > > > > > > > > > > > > > > > > > > > folks
> > >> > > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In
> order
> > to
> > >> > make
> > >> > > > > sure
> > >> > > > > > we
> > >> > > > > > > > > > > maintain
> > >> > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > deliver
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality
> > >> > pulsar-flink
> > >> > > > > > > > integration
> > >> > > > > > > > > > to
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > users
> > >> > > > > > > > > > > > > > > > > who
> > >> > > > > > > > > > > > > > > > > > > use
> > >> > > > > > > > > > > > > > > > > > > > > both
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we
> > need
> > >> > some
> > >> > > > help
> > >> > > > > > > from
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > > experts
> > >> > > > > > > > > > > > > > > from
> > >> > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > community.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been
> > >> following
> > >> > > > > FLIP-27
> > >> > > > > > > > for a
> > >> > > > > > > > > > > > while.
> > >> > > > > > > > > > > > > > > > > Originally
> > >> > > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > > were
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > thinking
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing
> the
> > >> > > > connectors
> > >> > > > > > back
> > >> > > > > > > > > after
> > >> > > > > > > > > > > > > > > integrating
> > >> > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced
> > in
> > >> > > > FLIP-27.
> > >> > > > > > But
> > >> > > > > > > we
> > >> > > > > > > > > > > decided
> > >> > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > initiate
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > conversation
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as
> > possible.
> > >> > > Because
> > >> > > > > we
> > >> > > > > > > > > believe
> > >> > > > > > > > > > > > there
> > >> > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > > > > > > benefits
> > >> > > > > > > > > > > > > > > > > > > > > > > > doing
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than
> > >> later.
> > >> > > As
> > >> > > > > part
> > >> > > > > > > of
> > >> > > > > > > > > > > > > contribution,
> > >> > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > can
> > >> > > > > > > > > > > > > > > > > > > help
> > >> > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community
> understand
> > >> more
> > >> > > > about
> > >> > > > > > > > Pulsar
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > potential
> > >> > > > > > > > > > > > > > > > > > > > > > > integration
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > points.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also
> > help
> > >> > Flink
> > >> > > > > > > community
> > >> > > > > > > > > > > verify
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > well
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API
> > (e.g.
> > >> > > > catalog
> > >> > > > > > > API).
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> 2019
> > at
> > >> > 5:24
> > >> > > > AM
> > >> > > > > > > Becket
> > >> > > > > > > > > > Qin <
> > >> > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> > >> interest
> > >> > > in
> > >> > > > > > > > > contributing
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > connector.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I
> > think
> > >> > > having
> > >> > > > > > Pulsar
> > >> > > > > > > > > > > connector
> > >> > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > strong
> > >> > > > > > > > > > > > > > > > > > > > > support
> > >> > > > > > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > > > > a
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable
> addition
> > to
> > >> > > Flink.
> > >> > > > > So
> > >> > > > > > I
> > >> > > > > > > am
> > >> > > > > > > > > > happy
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > shepherd
> > >> > > > > > > > > > > > > > > > > > > this
> > >> > > > > > > > > > > > > > > > > > > > > > > effort.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I
> would
> > >> also
> > >> > > > like
> > >> > > > > to
> > >> > > > > > > > > provide
> > >> > > > > > > > > > > > some
> > >> > > > > > > > > > > > > > > > context
> > >> > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > recent
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink
> > connectors
> > >> > > > > ecosystem.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way
> > Flink
> > >> > > > > maintains
> > >> > > > > > > its
> > >> > > > > > > > > > > > > connector has
> > >> > > > > > > > > > > > > > > > hit
> > >> > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > scalability
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > bar.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and
> more
> > >> > > > connectors
> > >> > > > > > > > coming
> > >> > > > > > > > > > into
> > >> > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > repo,
> > >> > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > > > > > > > facing a
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > few
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as
> > long
> > >> > > build
> > >> > > > > and
> > >> > > > > > > > > testing
> > >> > > > > > > > > > > > > time. To
> > >> > > > > > > > > > > > > > > > > > address
> > >> > > > > > > > > > > > > > > > > > > this
> > >> > > > > > > > > > > > > > > > > > > > > > > > > problem,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted
> to
> > do
> > >> > the
> > >> > > > > > > following:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the
> > >> > > connectors
> > >> > > > > > into
> > >> > > > > > > a
> > >> > > > > > > > > > > separate
> > >> > > > > > > > > > > > > > > > > repository.
> > >> > > > > > > > > > > > > > > > > > > This
> > >> > > > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to
> > >> > potential
> > >> > > > > > solution
> > >> > > > > > > > to
> > >> > > > > > > > > > > > shorten
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > build
> > >> > > > > > > > > > > > > > > > > > > time.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the
> > >> > > connectors
> > >> > > > > to
> > >> > > > > > > stay
> > >> > > > > > > > > as
> > >> > > > > > > > > > > > > ecosystem
> > >> > > > > > > > > > > > > > > > > > project
> > >> > > > > > > > > > > > > > > > > > > > > while
> > >> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > tries
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good
> > >> support
> > >> > > for
> > >> > > > > > > > > > functionality
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > compatibility
> > >> > > > > > > > > > > > > > > > > > > > > > > tests.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Robert
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to
> > create
> > >> a
> > >> > > > Flink
> > >> > > > > > > > > Ecosystem
> > >> > > > > > > > > > > > > project
> > >> > > > > > > > > > > > > > > > > website
> > >> > > > > > > > > > > > > > > > > > > and it
> > >> > > > > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > > > > > going
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some
> final
> > >> > > approval
> > >> > > > > > > > process.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above
> > >> > efforts,
> > >> > > it
> > >> > > > > > would
> > >> > > > > > > > be
> > >> > > > > > > > > > > great
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > first
> > >> > > > > > > > > > > > > > > > > see
> > >> > > > > > > > > > > > > > > > > > > if we
> > >> > > > > > > > > > > > > > > > > > > > > > can
> > >> > > > > > > > > > > > > > > > > > > > > > > > > have
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector
> > as
> > >> an
> > >> > > > > > ecosystem
> > >> > > > > > > > > > project
> > >> > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > great
> > >> > > > > > > > > > > > > > > > > > > support.
> > >> > > > > > > > > > > > > > > > > > > > > It
> > >> > > > > > > > > > > > > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how
> > the
> > >> > > Flink
> > >> > > > > > Pulsar
> > >> > > > > > > > > > > connector
> > >> > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > tested
> > >> > > > > > > > > > > > > > > > > > > > > currently
> > >> > > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > see
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > if
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn
> > >> something
> > >> > to
> > >> > > > > > > maintain
> > >> > > > > > > > it
> > >> > > > > > > > > > as
> > >> > > > > > > > > > > an
> > >> > > > > > > > > > > > > > > > ecosystem
> > >> > > > > > > > > > > > > > > > > > > project
> > >> > > > > > > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > > > > > > > good
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test
> > >> > > coverage.
> > >> > > > If
> > >> > > > > > the
> > >> > > > > > > > > > quality
> > >> > > > > > > > > > > > as
> > >> > > > > > > > > > > > > an
> > >> > > > > > > > > > > > > > > > > > ecosystem
> > >> > > > > > > > > > > > > > > > > > > > > > project
> > >> > > > > > > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > hard
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we
> > may
> > >> as
> > >> > > > well
> > >> > > > > > > adopt
> > >> > > > > > > > it
> > >> > > > > > > > > > > into
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > main
> > >> > > > > > > > > > > > > > > > > > repo.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another
> > ongoing
> > >> > > effort
> > >> > > > > is
> > >> > > > > > > > > FLIP-27
> > >> > > > > > > > > > > > where
> > >> > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > > making
> > >> > > > > > > > > > > > > > > > > > > > > > > changes
> > >> > > > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source
> > >> connector
> > >> > > > > > > architecture
> > >> > > > > > > > > and
> > >> > > > > > > > > > > > > > > interface.
> > >> > > > > > > > > > > > > > > > > This
> > >> > > > > > > > > > > > > > > > > > > > > change
> > >> > > > > > > > > > > > > > > > > > > > > > > will
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > likely
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10.
> > >> Therefore
> > >> > > > > timing
> > >> > > > > > > > wise,
> > >> > > > > > > > > if
> > >> > > > > > > > > > > we
> > >> > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > going
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > have
> > >> > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in
> main
> > >> > repo, I
> > >> > > > am
> > >> > > > > > > > > wondering
> > >> > > > > > > > > > if
> > >> > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > should
> > >> > > > > > > > > > > > > > > > > > hold
> > >> > > > > > > > > > > > > > > > > > > a
> > >> > > > > > > > > > > > > > > > > > > > > > little
> > >> > > > > > > > > > > > > > > > > > > > > > > > bit
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar
> > >> > connector
> > >> > > > > adapt
> > >> > > > > > to
> > >> > > > > > > > the
> > >> > > > > > > > > > new
> > >> > > > > > > > > > > > > > > interface
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > avoid
> > >> > > > > > > > > > > > > > > > > > > > > > > > shortly
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work?
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie
> (Becket)
> > Qin
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> > 2019
> > >> at
> > >> > > 4:32
> > >> > > > > PM
> > >> > > > > > > > > Chesnay
> > >> > > > > > > > > > > > > Schepler <
> > >> > > > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite
> > worried
> > >> > that
> > >> > > we
> > >> > > > > may
> > >> > > > > > > end
> > >> > > > > > > > > up
> > >> > > > > > > > > > > > > repeating
> > >> > > > > > > > > > > > > > > > > > history.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were
> > already
> > >> 2
> > >> > > > > attempts
> > >> > > > > > > at
> > >> > > > > > > > > > > > > contributing a
> > >> > > > > > > > > > > > > > > > > > pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > connector,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > both
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which
> failed
> > >> > because
> > >> > > > no
> > >> > > > > > > > > committer
> > >> > > > > > > > > > > was
> > >> > > > > > > > > > > > > > > getting
> > >> > > > > > > > > > > > > > > > > > > involved,
> > >> > > > > > > > > > > > > > > > > > > > > > > > despite
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor
> > >> opening a
> > >> > > > > > dedicated
> > >> > > > > > > > > > > > discussion
> > >> > > > > > > > > > > > > > > thread
> > >> > > > > > > > > > > > > > > > > > > about the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > contribution
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and
> > >> > getting
> > >> > > > > > several
> > >> > > > > > > > +1's
> > >> > > > > > > > > > > from
> > >> > > > > > > > > > > > > > > > > committers.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should
> really
> > >> make
> > >> > > > sure
> > >> > > > > > that
> > >> > > > > > > > if
> > >> > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > welcome/approve
> > >> > > > > > > > > > > > > > > > > > > such
> > >> > > > > > > > > > > > > > > > > > > > > a
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution
> it
> > >> will
> > >> > > > > actually
> > >> > > > > > > get
> > >> > > > > > > > > the
> > >> > > > > > > > > > > > > attention
> > >> > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > > > deserves.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm
> > >> inclined
> > >> > > to
> > >> > > > > > > > recommend
> > >> > > > > > > > > > > > > maintaining
> > >> > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > > > > > > > outside
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We
> could
> > >> link
> > >> > to
> > >> > > > it
> > >> > > > > > from
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > > > documentation
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > give
> > >> > > > > > > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the
> > upcoming
> > >> > page
> > >> > > > for
> > >> > > > > > > > sharing
> > >> > > > > > > > > > > > > artifacts
> > >> > > > > > > > > > > > > > > > among
> > >> > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > community
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > (what's
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of
> > that
> > >> > > > anyway?),
> > >> > > > > > > this
> > >> > > > > > > > > may
> > >> > > > > > > > > > > be a
> > >> > > > > > > > > > > > > > > better
> > >> > > > > > > > > > > > > > > > > > > option.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019
> > >> 10:16,
> > >> > > Till
> > >> > > > > > > > Rohrmann
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot
> > for
> > >> > > > starting
> > >> > > > > > > this
> > >> > > > > > > > > > > > discussion
> > >> > > > > > > > > > > > > > > > Yijie.
> > >> > > > > > > > > > > > > > > > > I
> > >> > > > > > > > > > > > > > > > > > > think
> > >> > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector
> > would
> > >> be
> > >> > a
> > >> > > > very
> > >> > > > > > > > > valuable
> > >> > > > > > > > > > > > > addition
> > >> > > > > > > > > > > > > > > > since
> > >> > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > becomes
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular
> > and
> > >> it
> > >> > > > would
> > >> > > > > > > > further
> > >> > > > > > > > > > > > expand
> > >> > > > > > > > > > > > > > > > Flink's
> > >> > > > > > > > > > > > > > > > > > > > > > > > > interoperability.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Also
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a
> project
> > >> > > > > perspective
> > >> > > > > > it
> > >> > > > > > > > > makes
> > >> > > > > > > > > > > > > sense for
> > >> > > > > > > > > > > > > > > > me
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > place
> > >> > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> downstream
> > >> > > project.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> > >> > > > concern/question
> > >> > > > > is
> > >> > > > > > > how
> > >> > > > > > > > > can
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > community
> > >> > > > > > > > > > > > > > > > > > > > > > > > maintain
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector?
> We
> > >> have
> > >> > > seen
> > >> > > > > in
> > >> > > > > > > the
> > >> > > > > > > > > past
> > >> > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > > > connectors
> > >> > > > > > > > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > > > > > some
> > >> > > > > > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > most
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively
> > >> developed
> > >> > > > > > components
> > >> > > > > > > > > > because
> > >> > > > > > > > > > > > > they
> > >> > > > > > > > > > > > > > > need
> > >> > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > be
> > >> > > > > > > > > > > > > > > > > > > > > kept
> > >> > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > sync
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external
> > system
> > >> and
> > >> > > > with
> > >> > > > > > > Flink.
> > >> > > > > > > > > > Given
> > >> > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > community
> > >> > > > > > > > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with
> > >> > > > maintaining,
> > >> > > > > > > > > improving
> > >> > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > evolving
> > >> > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > connector,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > I'm
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic
> > that
> > >> we
> > >> > > can
> > >> > > > > > > achieve
> > >> > > > > > > > > > this.
> > >> > > > > > > > > > > > > Hence,
> > >> > > > > > > > > > > > > > > +1
> > >> > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > > > > contributing
> > >> > > > > > > > > > > > > > > > > > > > > > > > > it
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > back
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep
> 4,
> > >> 2019
> > >> > > at
> > >> > > > > 2:03
> > >> > > > > > > AM
> > >> > > > > > > > > > Sijie
> > >> > > > > > > > > > > > Guo
> > >> > > > > > > > > > > > > <
> > >> > > > > > > > > > > > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was
> > the
> > >> > main
> > >> > > > > > driver
> > >> > > > > > > > > behind
> > >> > > > > > > > > > > > > > > FLINK-9641
> > >> > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > let
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > me
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more
> > context
> > >> > on
> > >> > > > > this.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641
> > and
> > >> > > > > FLINK-9168
> > >> > > > > > > was
> > >> > > > > > > > > > > created
> > >> > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > bringing
> > >> > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > source
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for
> > Flink.
> > >> > The
> > >> > > > > > > > integration
> > >> > > > > > > > > > was
> > >> > > > > > > > > > > > done
> > >> > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > 1.6.0.
> > >> > > > > > > > > > > > > > > > > > > > > > > We
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > sent out
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests
> > about a
> > >> > > year
> > >> > > > > ago
> > >> > > > > > > and
> > >> > > > > > > > we
> > >> > > > > > > > > > > ended
> > >> > > > > > > > > > > > > up
> > >> > > > > > > > > > > > > > > > > > > maintaining
> > >> > > > > > > > > > > > > > > > > > > > > > those
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for
> > >> Pulsar
> > >> > > > users
> > >> > > > > to
> > >> > > > > > > use
> > >> > > > > > > > > > Flink
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > process
> > >> > > > > > > > > > > > > > > > > > > event
> > >> > > > > > > > > > > > > > > > > > > > > > > streams
> > >> > > > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > ).
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > The
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6
> > integration
> > >> is
> > >> > > > > pretty
> > >> > > > > > > > simple
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > > > there
> > >> > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > no
> > >> > > > > > > > > > > > > > > > > > > schema
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past
> > >> year,
> > >> > we
> > >> > > > > have
> > >> > > > > > > > made a
> > >> > > > > > > > > > lot
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > changes
> > >> > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > brought
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar
> > schema as
> > >> > the
> > >> > > > > > > > first-class
> > >> > > > > > > > > > > > > citizen in
> > >> > > > > > > > > > > > > > > > > > Pulsar.
> > >> > > > > > > > > > > > > > > > > > > We
> > >> > > > > > > > > > > > > > > > > > > > > > also
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > integrated
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other
> > computing
> > >> > > > engines
> > >> > > > > > for
> > >> > > > > > > > > > > processing
> > >> > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > event
> > >> > > > > > > > > > > > > > > > > > > > > > streams
> > >> > > > > > > > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us
> to
> > >> > rethink
> > >> > > > how
> > >> > > > > > to
> > >> > > > > > > > > > > integrate
> > >> > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > best
> > >> > > > > > > > > > > > > > > > > > > > > > > > > way.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Then
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement
> > the
> > >> > > > > > pulsar-flink
> > >> > > > > > > > > > > > connectors
> > >> > > > > > > > > > > > > from
> > >> > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > ground
> > >> > > > > > > > > > > > > > > > > > > > > up
> > >> > > > > > > > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table
> > API
> > >> > and
> > >> > > > > > catalog
> > >> > > > > > > > API
> > >> > > > > > > > > as
> > >> > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > first-class
> > >> > > > > > > > > > > > > > > > > > > > > citizen
> > >> > > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> integration.
> > >> With
> > >> > > that
> > >> > > > > > being
> > >> > > > > > > > > said,
> > >> > > > > > > > > > > in
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > > > pulsar-flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> implementation,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can
> > register
> > >> > > > pulsar
> > >> > > > > > as a
> > >> > > > > > > > > flink
> > >> > > > > > > > > > > > > catalog
> > >> > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > query /
> > >> > > > > > > > > > > > > > > > > > > > > > > process
> > >> > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > event
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams
> using
> > >> > Flink
> > >> > > > SQL.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an
> > >> example
> > >> > > > about
> > >> > > > > > how
> > >> > > > > > > > to
> > >> > > > > > > > > > use
> > >> > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > a
> > >> > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > catalog:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has
> > also
> > >> > > > written a
> > >> > > > > > > blog
> > >> > > > > > > > > post
> > >> > > > > > > > > > > > > > > explaining
> > >> > > > > > > > > > > > > > > > > why
> > >> > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > > > > > > re-implement
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector
> > with
> > >> > Flink
> > >> > > > 1.9
> > >> > > > > > and
> > >> > > > > > > > > what
> > >> > > > > > > > > > > are
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > changes
> > >> > > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > > > made
> > >> > > > > > > > > > > > > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > new
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe
> > >> Pulsar
> > >> > is
> > >> > > > not
> > >> > > > > > > just
> > >> > > > > > > > a
> > >> > > > > > > > > > > simple
> > >> > > > > > > > > > > > > data
> > >> > > > > > > > > > > > > > > > sink
> > >> > > > > > > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > > > > > > > > source
> > >> > > > > > > > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > It
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually
> can
> > be
> > >> a
> > >> > > > fully
> > >> > > > > > > > > integrated
> > >> > > > > > > > > > > > > streaming
> > >> > > > > > > > > > > > > > > > > data
> > >> > > > > > > > > > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas
> (sink,
> > >> > source,
> > >> > > > > > > > > > schema/catalog
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > state).
> > >> > > > > > > > > > > > > > > > > > The
> > >> > > > > > > > > > > > > > > > > > > > > > > > combination
> > >> > > > > > > > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar
> > can
> > >> > > create
> > >> > > > a
> > >> > > > > > > great
> > >> > > > > > > > > > > > streaming
> > >> > > > > > > > > > > > > > > > > warehouse
> > >> > > > > > > > > > > > > > > > > > > > > > > architecture
> > >> > > > > > > > > > > > > > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > streaming-first,
> > >> > > > unified
> > >> > > > > > > data
> > >> > > > > > > > > > > > > processing.
> > >> > > > > > > > > > > > > > > > Since
> > >> > > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > > > > > > talking
> > >> > > > > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute
> > >> Pulsar
> > >> > > > > > > integration
> > >> > > > > > > > to
> > >> > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > here,
> > >> > > > > > > > > > > > > > > > we
> > >> > > > > > > > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > > > > > > also
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain,
> > >> improve
> > >> > > and
> > >> > > > > > evolve
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > integration
> > >> > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > help
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use
> both
> > >> Flink
> > >> > > and
> > >> > > > > > > Pulsar.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this
> > give
> > >> > you a
> > >> > > > bit
> > >> > > > > > > more
> > >> > > > > > > > > > > > background
> > >> > > > > > > > > > > > > > > about
> > >> > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> integration.
> > Let
> > >> > me
> > >> > > > know
> > >> > > > > > > what
> > >> > > > > > > > > are
> > >> > > > > > > > > > > your
> > >> > > > > > > > > > > > > > > > thoughts.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep
> > 3,
> > >> > 2019
> > >> > > at
> > >> > > > > > 11:54
> > >> > > > > > > > AM
> > >> > > > > > > > > > Yun
> > >> > > > > > > > > > > > > Tang <
> > >> > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see
> > that
> > >> > > Pulsar
> > >> > > > > > > becomes
> > >> > > > > > > > > > more
> > >> > > > > > > > > > > > and
> > >> > > > > > > > > > > > > more
> > >> > > > > > > > > > > > > > > > > > popular
> > >> > > > > > > > > > > > > > > > > > > > > > > recently
> > >> > > > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > very
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see
> more
> > >> > people
> > >> > > > > > willing
> > >> > > > > > > to
> > >> > > > > > > > > > > > > contribute to
> > >> > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > ecosystem.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any
> > >> > further
> > >> > > > > > > > discussion,
> > >> > > > > > > > > > > would
> > >> > > > > > > > > > > > > you
> > >> > > > > > > > > > > > > > > > please
> > >> > > > > > > > > > > > > > > > > > > give
> > >> > > > > > > > > > > > > > > > > > > > > some
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > explanation
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the
> > >> relationship
> > >> > > > > between
> > >> > > > > > > this
> > >> > > > > > > > > > > thread
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > current
> > >> > > > > > > > > > > > > > > > > > > > > existing
> > >> > > > > > > > > > > > > > > > > > > > > > > > JIRAs
> > >> > > > > > > > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1]
> > and
> > >> > sink
> > >> > > > [2]
> > >> > > > > > > > > > connector?
> > >> > > > > > > > > > > > > Will the
> > >> > > > > > > > > > > > > > > > > > > > > contribution
> > >> > > > > > > > > > > > > > > > > > > > > > > > > contains
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > part
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs
> or
> > >> > > totally
> > >> > > > > > > > different
> > >> > > > > > > > > > > > > > > > implementation?
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > ________________________________
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From:
> Yijie
> > >> Shen
> > >> > <
> > >> > > > > > > > > > > > > > > [hidden email]>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent:
> > Tuesday,
> > >> > > > > September
> > >> > > > > > 3,
> > >> > > > > > > > > 2019
> > >> > > > > > > > > > > > 13:57
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> > >> > > > > [hidden email]
> > >> > > > > > <
> > >> > > > > > > > > > > > > > > > [hidden email]
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject:
> > >> > [DISCUSS]
> > >> > > > > > > Contribute
> > >> > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > > > back
> > >> > > > > > > > > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink
> > >> > > Community!
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would
> > like to
> > >> > > open
> > >> > > > > the
> > >> > > > > > > > > > discussion
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > contributing
> > >> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector
> > [0]
> > >> > back
> > >> > > to
> > >> > > > > > > Flink.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief
> > >> > > > introduction
> > >> > > > > > to
> > >> > > > > > > > > Apache
> > >> > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache
> > >> Pulsar[1]
> > >> > > is a
> > >> > > > > > > > > > multi-tenant,
> > >> > > > > > > > > > > > > > > > > > > high-performance
> > >> > > > > > > > > > > > > > > > > > > > > > > > > distributed
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub
> > >> messaging
> > >> > > > > system.
> > >> > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > includes
> > >> > > > > > > > > > > > > > > > > > multiple
> > >> > > > > > > > > > > > > > > > > > > > > > features
> > >> > > > > > > > > > > > > > > > > > > > > > > > > such
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native
> > support
> > >> > for
> > >> > > > > > multiple
> > >> > > > > > > > > > > clusters
> > >> > > > > > > > > > > > > in a
> > >> > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > instance,
> > >> > > > > > > > > > > > > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> > >> > > > > geo-replication
> > >> > > > > > of
> > >> > > > > > > > > > > messages
> > >> > > > > > > > > > > > > across
> > >> > > > > > > > > > > > > > > > > > > clusters,
> > >> > > > > > > > > > > > > > > > > > > > > > very
> > >> > > > > > > > > > > > > > > > > > > > > > > > low
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and
> > end-to-end
> > >> > > > latency,
> > >> > > > > > > > > seamless
> > >> > > > > > > > > > > > > > > scalability
> > >> > > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > over a
> > >> > > > > > > > > > > > > > > > > > > > > > > > million
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and
> > guaranteed
> > >> > > > message
> > >> > > > > > > > delivery
> > >> > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > persistent
> > >> > > > > > > > > > > > > > > > > > > message
> > >> > > > > > > > > > > > > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided
> by
> > >> > Apache
> > >> > > > > > > > BookKeeper.
> > >> > > > > > > > > > > > > Nowadays,
> > >> > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > has
> > >> > > > > > > > > > > > > > > > > > > > > been
> > >> > > > > > > > > > > > > > > > > > > > > > > > > adopted
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > by
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and
> > more
> > >> > > > > > companies[2].
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The
> > status
> > >> of
> > >> > > > Pulsar
> > >> > > > > > > Flink
> > >> > > > > > > > > > > > connector
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar
> > >> Flink
> > >> > > > > > connector
> > >> > > > > > > we
> > >> > > > > > > > > are
> > >> > > > > > > > > > > > > planning
> > >> > > > > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > > > > > > > > contribute
> > >> > > > > > > > > > > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > > > > > > > > > > > > built
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink
> 1.9.0
> > and
> > >> > > > Pulsar
> > >> > > > > > > 2.4.0.
> > >> > > > > > > > > The
> > >> > > > > > > > > > > > main
> > >> > > > > > > > > > > > > > > > features
> > >> > > > > > > > > > > > > > > > > > > are:
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar
> as
> > a
> > >> > > > streaming
> > >> > > > > > > > source
> > >> > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > exactly-once
> > >> > > > > > > > > > > > > > > > > > > > > > guarantee.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink
> > >> streaming
> > >> > > > > results
> > >> > > > > > to
> > >> > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > > > > > at-least-once
> > >> > > > > > > > > > > > > > > > > > > > > > > > > semantics.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > (We
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would
> update
> > >> this
> > >> > > to
> > >> > > > > > > > > exactly-once
> > >> > > > > > > > > > > as
> > >> > > > > > > > > > > > > well
> > >> > > > > > > > > > > > > > > > when
> > >> > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > gets
> > >> > > > > > > > > > > > > > > > > > > > > > > > all
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> transaction
> > >> > > features
> > >> > > > > > ready
> > >> > > > > > > in
> > >> > > > > > > > > its
> > >> > > > > > > > > > > > 2.5.0
> > >> > > > > > > > > > > > > > > > > version)
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build
> upon
> > >> > Flink
> > >> > > > new
> > >> > > > > > > Table
> > >> > > > > > > > > API
> > >> > > > > > > > > > > Type
> > >> > > > > > > > > > > > > > > system
> > >> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > >> > > > > > > > > > > > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > can
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > automatically
> > >> > > > > > (de)serialize
> > >> > > > > > > > > > > messages
> > >> > > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > help
> > >> > > > > > > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> -
> Integrate
> > >> with
> > >> > > > Flink
> > >> > > > > > new
> > >> > > > > > > > > > Catalog
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > > > > > > (FLIP-30[4]),
> > >> > > > > > > > > > > > > > > > > > > > > > which
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > enables
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of
> > Pulsar
> > >> > > topics
> > >> > > > as
> > >> > > > > > > > tables
> > >> > > > > > > > > in
> > >> > > > > > > > > > > > > Table API
> > >> > > > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > > > > > > > > well as
> > >> > > > > > > > > > > > > > > > > > > > > > SQL
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > client.
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ##
> Reference
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > >> > > > > > > > > > > > > > > > >
> > >> https://github.com/streamnative/pulsar-flink
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > >> > > > > > > > https://pulsar.apache.org/
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > >> > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>

[Jark Wu-2]

I agree that we can start from Pulsar sink and Pulsar Catalog.
When we finish that, maybe FLIP-27 is ready then.

Best,
Jark

On Tue, 24 Sep 2019 at 17:06, Becket Qin <[hidden email]> wrote:

> Hi Till,
>
> My understanding is that we are going to hold back a little bit on the
> Pulsar source connector until FLIP-27 is ready. If we check in the old
> Pulsar source connector right away, we will have to maintain that old
> connector for quite a while (a year or more), which is something we don't
> want to do.
>
> So the conclusion is following:
> - We will try our best to check in both FLIP-27 and the Pulsar source
> connector in 1.10.
> - In case there isn't enough time to get Pulsar source connector checked in
> 1.10, we will link prominently to the external Pulsar connector repo in our
> connector doc so that users can find that easily.
>
> BTW, I think we can start the work on Pulsar sink connector given that
> there is no clear plan on whether or how we are going to change the sink
> interface.
>
> @Sijie, @Yijie, what do you think?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Tue, Sep 24, 2019 at 4:40 PM Till Rohrmann <[hidden email]>
> wrote:
>
> > What is the conclusion of this discussion? Are we gonna merge the old
> > source/sink or do we believe that FLIP-27 will be completed in time so
> that
> > we can merge the new source right away?
> >
> > Cheers,
> > Till
> >
> > On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <[hidden email]> wrote:
> >
> > >
> > >
> > > unscribe
> > >
> > >
> > >
> > >
> > >
> > >
> > > At 2019-09-24 08:53:25, "Becket Qin" <[hidden email]> wrote:
> > > >Thanks, Stephan.
> > > >
> > > >Sounds good to me. We can still try our best to get new Pulsar
> connector
> > > in
> > > >Flink 1.10. In case we do not have time to do that, we will
> prominently
> > > >link the Pulsar connector from the Flink connector docs.
> > > >
> > > >Thanks,
> > > >
> > > >Jiangjie (Becket) Qin
> > > >
> > > >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <[hidden email]>
> wrote:
> > > >
> > > >> Okay, I see your point, Becket.
> > > >>
> > > >> Then let us prominently link the Pulsar connector from the Flink
> > > connector
> > > >> docs then, so that users can find it easily.
> > > >>
> > > >> As soon as FLIP 27 is done, we reach out the Pulsar folks to
> > contribute
> > > a
> > > >> new connector.
> > > >>
> > > >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]>
> > > wrote:
> > > >>
> > > >> > Hi Stephan,
> > > >> >
> > > >> > I have no doubt about the value of adding Pulsar connector to
> Flink
> > > repo.
> > > >> > My concern is about how exactly we are going to do it.
> > > >> >
> > > >> > As mentioned before, I believe that we can handle connectors more
> > > >> > > pragmatically and less strict than the core of Flink, if it
> helps
> > > >> > unlocking
> > > >> > > users.
> > > >> >
> > > >> > I can see the benefit of being less restrict for the initial
> > connector
> > > >> code
> > > >> > adoption. However, I don't think we should be less restrict on the
> > > >> > maintenance commitment once the code is in Flink repo. It only
> makes
> > > >> sense
> > > >> > to check in something and ask users to use if we plan to maintain
> > it.
> > > >> >
> > > >> > If I understand correctly, the current plan so far is following:
> > > >> > 1. release 1.10
> > > >> >    - Check in Pulsar connector on old interface and label it as
> beta
> > > >> > version.
> > > >> >    - encourage users to try it and report bugs.
> > > >> > 2. release 1.11
> > > >> >    - Check in Pulsar connector on new interface (a.k.a new Pulsar
> > > >> > connector) and label it as beta version
> > > >> >    - Deprecate the old Pulsar connector
> > > >> >    - Fix bugs reported on old Pulsar connector from release 1.10
> > > >> >    - Ask users to migrate from old Pulsar connector to new Pulsar
> > > >> connector
> > > >> > 3. release 1.12
> > > >> >    - Announce end of support for old Pulsar connector and remove
> the
> > > code
> > > >> >    - Fix bugs reported on new Pulsar connector.
> > > >> >
> > > >> > If this is the plan, it seems neither Flink nor the users trying
> the
> > > old
> > > >> > Pulsar connector will benefit from this experimental old Pulsar
> > > >> connector,
> > > >> > because whatever feedbacks we got or bugs we fix on the old Pulsar
> > > >> > connector are immediately thrown away in one or two releases.
> > > >> >
> > > >> > If we check in the old Pulsar connector right now, the only
> option I
> > > see
> > > >> is
> > > >> > to maintain it for a while (e.g. a year or more). IMO, the
> immediate
> > > >> > deprecation and code removal hurts the users much more than asking
> > > them
> > > >> to
> > > >> > wait for another release. I personally think that we can avoid
> this
> > > >> > maintenance burden by going directly to the new Pulsar connector,
> > > >> > especially given that users can still use the connector even if
> they
> > > are
> > > >> > not in Flink repo. That said, I am OK with maintaining both old
> and
> > > new
> > > >> > Pulsar connector if we believe that having the Pulsar connector
> > > available
> > > >> > right now in Flink repo is more important.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jiangjie (Becket) Qin
> > > >> >
> > > >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]>
> > > wrote:
> > > >> >
> > > >> > > My assumption is as Sijie's, that once the connector is either
> > part
> > > of
> > > >> > > Flink, or part of the streamnative repo. No double maintenance.
> > > >> > >
> > > >> > > I feel this discussion is very much caught in problems that are
> > all
> > > >> > > solvable if we want to solve them.
> > > >> > > Maybe we can think what our goal for users and the communities
> is?
> > > >> > >
> > > >> > >   - Do we want to help build a relationship between the Pulsar
> and
> > > >> Flink
> > > >> > > open source communities?
> > > >> > >   - Will users find a connector in the streamnative repository?
> > > >> > >   - Will users trust a connector that is not part of Flink as
> > much?
> > > >> > >
> > > >> > > And then decide what is best according to the overall goals
> there.
> > > >> > > As mentioned before, I believe that we can handle connectors
> more
> > > >> > > pragmatically and less strict than the core of Flink, if it
> helps
> > > >> > unlocking
> > > >> > > users.
> > > >> > >
> > > >> > > Best,
> > > >> > > Stephan
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]>
> > > wrote:
> > > >> > >
> > > >> > > > Thanks Becket.
> > > >> > > >
> > > >> > > > I think it is better for the Flink community to judge the
> > > benefits of
> > > >> > > doing
> > > >> > > > this. I was trying to provide some views from outsiders.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Sijie
> > > >> > > >
> > > >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <
> > [hidden email]
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hi Sijie,
> > > >> > > > >
> > > >> > > > > Yes, we will have to support existing old connectors and new
> > > >> > connectors
> > > >> > > > in
> > > >> > > > > parallel for a while. We have to take that maintenance
> > overhead
> > > >> > because
> > > >> > > > > existing connectors have been used by the users for a long
> > > time. I
> > > >> > > guess
> > > >> > > > It
> > > >> > > > > may take at least a year for us to fully remove the old
> > > connectors.
> > > >> > > > >
> > > >> > > > > Process wise, we can do the same for Pulsar connector. But I
> > am
> > > not
> > > >> > > sure
> > > >> > > > if
> > > >> > > > > we want to have the same burden on Pulsar connector, and I
> > would
> > > >> like
> > > >> > > to
> > > >> > > > > understand the benefit of doing that.
> > > >> > > > >
> > > >> > > > > For users, the benefit of having the old Pulsar connector
> > > checked
> > > >> in
> > > >> > > > seems
> > > >> > > > > limited because 1) that code base will be immediately
> > > deprecated in
> > > >> > the
> > > >> > > > > next release in 3-4 months; 2) users can always use it even
> if
> > > it
> > > >> is
> > > >> > > not
> > > >> > > > in
> > > >> > > > > the Flink code base. Admittedly it is not as convenient as
> > > having
> > > >> it
> > > >> > in
> > > >> > > > > Flink code base, but doesn't seem super either. And after
> 3-4
> > > >> months,
> > > >> > > > users
> > > >> > > > > can just use the new connector in Flink repo.
> > > >> > > > >
> > > >> > > > > For Flink developers, the old connector code base is not
> > > something
> > > >> > that
> > > >> > > > we
> > > >> > > > > want to evolve later. Instead, these code will be deprecated
> > and
> > > >> > > > > removed. So why do we want to get a beta version out to
> > attract
> > > >> > people
> > > >> > > to
> > > >> > > > > use something we don't want to maintain?
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > >
> > > >> > > > > Jiangjie (Becket) Qin
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <
> > [hidden email]>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > Thanks everyone here. Sorry for jumping into the
> discussion
> > > here.
> > > >> > > > > >
> > > >> > > > > > I am not very familiar about the deprecation process in
> > Flink.
> > > >> If I
> > > >> > > > > > misunderstood the process, please fix me.
> > > >> > > > > >
> > > >> > > > > > As far as I understand, FLIP-27 is introducing a new
> unified
> > > API
> > > >> > for
> > > >> > > > > > connectors. After it introduces the new API
> > > >> > > > > > and before moving all the existing connectors from old API
> > to
> > > new
> > > >> > > API,
> > > >> > > > > both
> > > >> > > > > > old ApI and new API will co-exist
> > > >> > > > > > for a while until Flink moves all existing connectors to
> new
> > > API.
> > > >> > So
> > > >> > > > the
> > > >> > > > > > Pulsar connector (using old API) can
> > > >> > > > > > follow the deprecation process with other connector using
> > old
> > > API
> > > >> > and
> > > >> > > > the
> > > >> > > > > > deprecation of old API, no?
> > > >> > > > > >
> > > >> > > > > > If that's the case, I think contributing the current
> > connector
> > > >> back
> > > >> > > to
> > > >> > > > > > Flink rather than maintaining it outside Flink
> > > >> > > > > > would provide a bit more benefits. We can deprecate the
> > > existing
> > > >> > > > > > streamnative/pulsar-flink repo and point the users
> > > >> > > > > > to use the connector in Flink repo. So all the review
> > > processes
> > > >> > will
> > > >> > > > > happen
> > > >> > > > > > within Flink for both old connector and
> > > >> > > > > > new connector. It also reduces the confusions for the
> users
> > as
> > > >> the
> > > >> > > > > > documentation and code base happen in one place.
> > > >> > > > > >
> > > >> > > > > > Thoughts?
> > > >> > > > > >
> > > >> > > > > > - Sijie
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <
> > > >> [hidden email]>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Thanks for the explanation, Stephan. I have a few
> > questions
> > > /
> > > >> > > > thoughts.
> > > >> > > > > > >
> > > >> > > > > > > So that means we will remove the old connector without a
> > > major
> > > >> > > > version
> > > >> > > > > > > bump, is that correct?
> > > >> > > > > > >
> > > >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11
> > > >> connectors
> > > >> > > > will
> > > >> > > > > > > always work because we saw some dependency class
> > collisions
> > > in
> > > >> > the
> > > >> > > > > past.
> > > >> > > > > > To
> > > >> > > > > > > make it safe we may have to maintain the old code for
> one
> > > more
> > > >> > > > release.
> > > >> > > > > > >
> > > >> > > > > > > To be honest I am still wondering if we have to put the
> > old
> > > >> > > connector
> > > >> > > > > in
> > > >> > > > > > > Flink repo. if we check in the old connector to Flink.
> We
> > > will
> > > >> > end
> > > >> > > up
> > > >> > > > > in
> > > >> > > > > > > the following situation:
> > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > >> > > > > > > 2. Old connector in Flink Repo, which may be different
> > from
> > > the
> > > >> > one
> > > >> > > > in
> > > >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed
> > in
> > > >> 1.12)
> > > >> > > > > > > 3. New connector in Flink Repo.
> > > >> > > > > > >
> > > >> > > > > > > We need to think about how to make the users in each
> case
> > > >> happy.
> > > >> > > > > > > - For users of (1), I assume Sijie and Yijie will have
> to
> > > >> > maintain
> > > >> > > > the
> > > >> > > > > > code
> > > >> > > > > > > a bit longer for its own compatibility even after we
> have
> > > (2).
> > > >> In
> > > >> > > > that
> > > >> > > > > > > case, bugs found in old connector may or may not need to
> > be
> > > >> fixed
> > > >> > > in
> > > >> > > > > both
> > > >> > > > > > > Flink and the streamnative/pulsar-flink repo.
> > > >> > > > > > > - For users of (2), will we provide bug fixes? If we do,
> > it
> > > >> will
> > > >> > > be a
> > > >> > > > > > > little awkward because those bug fixes will be
> immediately
> > > >> > > deprecated
> > > >> > > > > in
> > > >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking
> > > users
> > > >> to
> > > >> > > > > migrate
> > > >> > > > > > > away from the bug fix. After Flink 1.12, users may still
> > > have
> > > >> to
> > > >> > > > switch
> > > >> > > > > > to
> > > >> > > > > > > use (3) due to the potential dependency class conflicts
> > > >> mentioned
> > > >> > > > > above.
> > > >> > > > > > > - Users of (3) have a much easier life and don't need to
> > > worry
> > > >> > too
> > > >> > > > > much.
> > > >> > > > > > >
> > > >> > > > > > > The above story seems a little complicated to tell. I
> > think
> > > it
> > > >> > will
> > > >> > > > be
> > > >> > > > > > much
> > > >> > > > > > > easier to not have (2) at all.
> > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
> > > >> > > > > > > 3. New connector in Flink Repo.
> > > >> > > > > > >
> > > >> > > > > > > - Old connector will only be maintained in
> > > >> > > streamnative/pulsar-flink
> > > >> > > > > repo
> > > >> > > > > > > until it is fully deprecated. Users can always use the
> > > existing
> > > >> > > > Pulsar
> > > >> > > > > > > connector in that repo.
> > > >> > > > > > > - New connector will be in Flink repo and maintained
> like
> > > the
> > > >> > other
> > > >> > > > > > > connectors.
> > > >> > > > > > >
> > > >> > > > > > > This seems much simpler for users understand and they
> will
> > > not
> > > >> be
> > > >> > > > > blocked
> > > >> > > > > > > from using the old connector. If the concern is about
> the
> > > >> quality
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > > connector in streamnative/pulsar-flink repo, is it
> enough
> > > for
> > > >> us
> > > >> > > just
> > > >> > > > > to
> > > >> > > > > > > review the code in streamnative/pulsar-flink connector
> to
> > > make
> > > >> > sure
> > > >> > > > it
> > > >> > > > > > > looks good from Flink's perspective?
> > > >> > > > > > >
> > > >> > > > > > > What do you think?
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > >
> > > >> > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <
> > > [hidden email]
> > > >> >
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > My take would be the following:
> > > >> > > > > > > >
> > > >> > > > > > > >   - If we merge the connector now and replace it with
> a
> > > >> FLIP-27
> > > >> > > > > version
> > > >> > > > > > > > before the 1.10 release, then we need no deprecation
> > > process
> > > >> > > > > > > >   - If we don't manage to replace it with a FLIP-27
> > > version
> > > >> > > before
> > > >> > > > > the
> > > >> > > > > > > 1.10
> > > >> > > > > > > > release, than it is good that we have the other
> version,
> > > so
> > > >> no
> > > >> > > > users
> > > >> > > > > > get
> > > >> > > > > > > > blocked.
> > > >> > > > > > > >
> > > >> > > > > > > > In the latter case we can see how we want to do it.
> > > Immediate
> > > >> > > > removal
> > > >> > > > > > of
> > > >> > > > > > > > the old version or deprecation label and keeping it
> for
> > > one
> > > >> > more
> > > >> > > > > > release.
> > > >> > > > > > > > Given that you should be able to use a Flink 1.10
> > > connector
> > > >> > with
> > > >> > > > > Flink
> > > >> > > > > > > 1.11
> > > >> > > > > > > > as well (stable public APIs) there is also a
> workaround
> > if
> > > >> you
> > > >> > > need
> > > >> > > > > an
> > > >> > > > > > > old
> > > >> > > > > > > > connector in a newer version. So immediate removal
> might
> > > even
> > > >> > be
> > > >> > > > > > > feasible.
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
> > > >> > > [hidden email]>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Stephan,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the clarification. I completely agree
> with
> > > you
> > > >> and
> > > >> > > > > Thomas
> > > >> > > > > > on
> > > >> > > > > > > > the
> > > >> > > > > > > > > process of adding connectors to Flink repo.
> However, I
> > > am
> > > >> > > > wondering
> > > >> > > > > > > what
> > > >> > > > > > > > is
> > > >> > > > > > > > > the deprecation process? Given the main concern here
> > was
> > > >> that
> > > >> > > we
> > > >> > > > > may
> > > >> > > > > > > have
> > > >> > > > > > > > > to maintain two Pulsar connector code bases until
> the
> > > old
> > > >> one
> > > >> > > is
> > > >> > > > > > > removed
> > > >> > > > > > > > > from the repo, it would be good to know how long we
> > > have to
> > > >> > do
> > > >> > > > > that.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
> > > >> > [hidden email]
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Some quick thoughts on the connector contribution
> > > >> process.
> > > >> > I
> > > >> > > > > > > basically
> > > >> > > > > > > > > > reiterate here what Thomas mentioned in another
> > thread
> > > >> > about
> > > >> > > > the
> > > >> > > > > > > > Kinesis
> > > >> > > > > > > > > > connector.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > For connectors, we should favor a low-overhead
> > > >> contribution
> > > >> > > > > > process,
> > > >> > > > > > > > and
> > > >> > > > > > > > > > accept user code and changes more readily than in
> > the
> > > >> core
> > > >> > > > > system.
> > > >> > > > > > > > > > That is because connectors have both a big variety
> > of
> > > >> > > scenarios
> > > >> > > > > > they
> > > >> > > > > > > > get
> > > >> > > > > > > > > > used in (only through use and many small
> > > contributions do
> > > >> > > they
> > > >> > > > > > become
> > > >> > > > > > > > > > really useful over time) and at the same time, and
> > > >> > committers
> > > >> > > > do
> > > >> > > > > > not
> > > >> > > > > > > > use
> > > >> > > > > > > > > > the connector themselves and usually cannot
> foresee
> > > too
> > > >> > well
> > > >> > > > what
> > > >> > > > > > is
> > > >> > > > > > > > > > needed.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Further more, a missing connector (or connector
> > > feature)
> > > >> is
> > > >> > > > > often a
> > > >> > > > > > > > > bigger
> > > >> > > > > > > > > > show stopper for users than a missing API or
> system
> > > >> > feature.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Along these lines of thougt, the conclusion would
> be
> > > to
> > > >> > take
> > > >> > > > the
> > > >> > > > > > > Pulsar
> > > >> > > > > > > > > > connector now, focus the review on
> > > >> legal/dependencies/rough
> > > >> > > > code
> > > >> > > > > > > style
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > conventions, label it as "beta" (in the sense of
> > "new
> > > >> code"
> > > >> > > > that
> > > >> > > > > is
> > > >> > > > > > > > "not
> > > >> > > > > > > > > > yet tested through longer use") and go ahead. And
> > then
> > > >> > evolve
> > > >> > > > it
> > > >> > > > > > > > quickly
> > > >> > > > > > > > > > without putting formal blockers in the way,
> meaning
> > > also
> > > >> > > > adding a
> > > >> > > > > > new
> > > >> > > > > > > > > FLIP
> > > >> > > > > > > > > > 27 version when it is there.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Best,
> > > >> > > > > > > > > > Stephan
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
> > > >> > > > [hidden email]
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Hi Yijie,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Could you please follow the FLIP process to
> start
> > a
> > > new
> > > >> > > FLIP
> > > >> > > > > > > > > [DISCUSSION]
> > > >> > > > > > > > > > > thread in the mailing list?
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list
> > > now.
> > > >> So
> > > >> > > > there
> > > >> > > > > > is a
> > > >> > > > > > > > > FLIP
> > > >> > > > > > > > > > > number collision. Can you change the FLIP number
> > to
> > > 72?
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
> > > >> > > > > [hidden email]>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Hi Yijie,
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP.
> > > >> > > > > > > > > > > > Would you mind enabling comments/suggestions
> on
> > > the
> > > >> > > google
> > > >> > > > > doc
> > > >> > > > > > > > link?
> > > >> > > > > > > > > > This
> > > >> > > > > > > > > > > > way the contributors from the community can
> > > comment
> > > >> on
> > > >> > > the
> > > >> > > > > doc.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > Rong
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
> > > >> > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Hello everyone,
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > I've drafted a FLIP that describes the
> current
> > > >> design
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > connector:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Please take a look and let me know what you
> > > think.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > Yijie
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong <
> > > >> > > > > > > [hidden email]>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi All,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Sorry for joining the discussion late and
> > > thanks
> > > >> > > Yijie
> > > >> > > > &
> > > >> > > > > > > Sijie
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > > > driving
> > > >> > > > > > > > > > > > > > the discussion.
> > > >> > > > > > > > > > > > > > I also think the Pulsar connector would
> be a
> > > very
> > > >> > > > > valuable
> > > >> > > > > > > > > addition
> > > >> > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > Flink. I can also help out a bit on the
> > review
> > > >> side
> > > >> > > :-)
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Regarding the timeline, I also share
> > concerns
> > > >> with
> > > >> > > > Becket
> > > >> > > > > > on
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > > relationship between the new Pulsar
> > connector
> > > and
> > > >> > > > > FLIP-27.
> > > >> > > > > > > > > > > > > > There's also another discussion just
> started
> > > by
> > > >> > > Stephan
> > > >> > > > > on
> > > >> > > > > > > > > dropping
> > > >> > > > > > > > > > > > Kafka
> > > >> > > > > > > > > > > > > > 9/10 support on next Flink release [1].
> > > Although
> > > >> > the
> > > >> > > > > > > situation
> > > >> > > > > > > > > is
> > > >> > > > > > > > > > > > > somewhat
> > > >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has
> been
> > > in
> > > >> > Flink
> > > >> > > > for
> > > >> > > > > > > > almost
> > > >> > > > > > > > > > 3-4
> > > >> > > > > > > > > > > > > years,
> > > >> > > > > > > > > > > > > > based on the discussion I am not sure if a
> > > major
> > > >> > > > version
> > > >> > > > > > > > release
> > > >> > > > > > > > > > is a
> > > >> > > > > > > > > > > > > > requirement for removing old connector
> > > supports.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I think there shouldn't be a blocker if we
> > > agree
> > > >> > the
> > > >> > > > old
> > > >> > > > > > > > > connector
> > > >> > > > > > > > > > > will
> > > >> > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar
> connector
> > is
> > > >> > there.
> > > >> > > > As
> > > >> > > > > > > > Stephan
> > > >> > > > > > > > > > > > stated,
> > > >> > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > is easier to contribute the source sooner
> > and
> > > >> > adjust
> > > >> > > it
> > > >> > > > > > > later.
> > > >> > > > > > > > > > > > > > We should also ensure we clearly
> communicate
> > > the
> > > >> > > > message:
> > > >> > > > > > for
> > > >> > > > > > > > > > > example,
> > > >> > > > > > > > > > > > > > putting an experimental flag on the
> > pre-FLIP27
> > > >> > > > connector
> > > >> > > > > > page
> > > >> > > > > > > > of
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > website, documentations, etc. Any other
> > > thoughts?
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > --
> > > >> > > > > > > > > > > > > > Rong
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > [1]
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket
> Qin <
> > > >> > > > > > > > [hidden email]
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Technically speaking, removing the old
> > > >> connector
> > > >> > > code
> > > >> > > > > is
> > > >> > > > > > a
> > > >> > > > > > > > > > > backwards
> > > >> > > > > > > > > > > > > > > incompatible change which requires a
> major
> > > >> > version
> > > >> > > > > bump,
> > > >> > > > > > > i.e.
> > > >> > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > 2.x.
> > > >> > > > > > > > > > > > > > > Given that we don't have a clear plan on
> > > when
> > > >> to
> > > >> > > have
> > > >> > > > > the
> > > >> > > > > > > > next
> > > >> > > > > > > > > > > major
> > > >> > > > > > > > > > > > > > > version release, it seems unclear how
> long
> > > the
> > > >> > old
> > > >> > > > > > > connector
> > > >> > > > > > > > > code
> > > >> > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > there if we check it in right now. Or
> will
> > > we
> > > >> > > remove
> > > >> > > > > the
> > > >> > > > > > > old
> > > >> > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > without a major version bump? In any
> case,
> > > it
> > > >> > > sounds
> > > >> > > > > not
> > > >> > > > > > > > quite
> > > >> > > > > > > > > > user
> > > >> > > > > > > > > > > > > > > friendly to the those who might use the
> > old
> > > >> > Pulsar
> > > >> > > > > > > > connector. I
> > > >> > > > > > > > > > am
> > > >> > > > > > > > > > > > not
> > > >> > > > > > > > > > > > > sure
> > > >> > > > > > > > > > > > > > > if it is worth these potential problems
> in
> > > >> order
> > > >> > to
> > > >> > > > > have
> > > >> > > > > > > the
> > > >> > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > connector checked in one or two months
> > > earlier.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan
> > > Ewen <
> > > >> > > > > > > > [hidden email]
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > Agreed, if we check in the old code,
> we
> > > >> should
> > > >> > > make
> > > >> > > > > it
> > > >> > > > > > > > clear
> > > >> > > > > > > > > > that
> > > >> > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27
> based
> > > >> version
> > > >> > > of
> > > >> > > > > the
> > > >> > > > > > > > > > connector
> > > >> > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > there.
> > > >> > > > > > > > > > > > > > > > We should not commit to maintaining
> the
> > > old
> > > >> > > > version,
> > > >> > > > > > that
> > > >> > > > > > > > > would
> > > >> > > > > > > > > > > be
> > > >> > > > > > > > > > > > > indeed
> > > >> > > > > > > > > > > > > > > > too much overhead.
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket
> > > Qin <
> > > >> > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Hi Stephan,
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Thanks for the volunteering to help.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Yes, the overhead would just be
> review
> > > >> > > capacity.
> > > >> > > > In
> > > >> > > > > > > > fact, I
> > > >> > > > > > > > > > am
> > > >> > > > > > > > > > > > not
> > > >> > > > > > > > > > > > > > > > worrying
> > > >> > > > > > > > > > > > > > > > > too much about the review capacity.
> > > That is
> > > >> > > just
> > > >> > > > a
> > > >> > > > > > one
> > > >> > > > > > > > time
> > > >> > > > > > > > > > > cost.
> > > >> > > > > > > > > > > > > My
> > > >> > > > > > > > > > > > > > > > > concern is mainly about the long
> term
> > > >> burden.
> > > >> > > > > Assume
> > > >> > > > > > we
> > > >> > > > > > > > > have
> > > >> > > > > > > > > > > new
> > > >> > > > > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly
> > added
> > > >> > Pulsar
> > > >> > > > > > > > connectors
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > old
> > > >> > > > > > > > > > > > > > > > > interface. Later on if we migrate
> > > Pulsar to
> > > >> > new
> > > >> > > > > > source
> > > >> > > > > > > > > > > interface,
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > old
> > > >> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated
> > > almost
> > > >> > > > > > immediately
> > > >> > > > > > > > > after
> > > >> > > > > > > > > > > > > checked
> > > >> > > > > > > > > > > > > > > in,
> > > >> > > > > > > > > > > > > > > > > but we may still have to maintain
> two
> > > code
> > > >> > > bases.
> > > >> > > > > For
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > existing
> > > >> > > > > > > > > > > > > > > > > connectors, we have to do that
> > anyways.
> > > But
> > > >> > it
> > > >> > > > > would
> > > >> > > > > > be
> > > >> > > > > > > > > good
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > > avoid
> > > >> > > > > > > > > > > > > > > > > introducing a new connector with the
> > > same
> > > >> > > > problem.
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM
> > Stephan
> > > >> Ewen
> > > >> > <
> > > >> > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Hi all!
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Nice to see this lively discussion
> > > about
> > > >> > the
> > > >> > > > > Pulsar
> > > >> > > > > > > > > > > connector.
> > > >> > > > > > > > > > > > > > > > > > Some thoughts on the open
> questions:
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain
> > as
> > > a
> > > >> > > > community
> > > >> > > > > > > > package
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Looks like the discussion is more
> > > going
> > > >> > > towards
> > > >> > > > > > > > > > > contribution. I
> > > >> > > > > > > > > > > > > think
> > > >> > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > is good, especially if we think
> that
> > > we
> > > >> > want
> > > >> > > to
> > > >> > > > > > > build a
> > > >> > > > > > > > > > > > similarly
> > > >> > > > > > > > > > > > > > > deep
> > > >> > > > > > > > > > > > > > > > > > integration with Pulsar as we have
> > for
> > > >> > > example
> > > >> > > > > with
> > > >> > > > > > > > > Kafka.
> > > >> > > > > > > > > > > The
> > > >> > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > already looks like a more thorough
> > > >> > connector
> > > >> > > > than
> > > >> > > > > > > many
> > > >> > > > > > > > > > others
> > > >> > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > have
> > > >> > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > the repository.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > With either a repo split, or the
> new
> > > >> build
> > > >> > > > > system,
> > > >> > > > > > I
> > > >> > > > > > > > hope
> > > >> > > > > > > > > > > that
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > build
> > > >> > > > > > > > > > > > > > > > > > overhead is not a problem.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > ## Committer Support
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Becket offered some help already,
> I
> > > can
> > > >> > also
> > > >> > > > > help a
> > > >> > > > > > > > bit.
> > > >> > > > > > > > > I
> > > >> > > > > > > > > > > hope
> > > >> > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > between us, we can cover this.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for
> > FLIP-27
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is
> actually
> > > >> making
> > > >> > > some
> > > >> > > > > > > PoC-ing
> > > >> > > > > > > > > > > > > progress, but
> > > >> > > > > > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > > > take 2 more months, I would
> > estimate,
> > > >> > before
> > > >> > > it
> > > >> > > > > is
> > > >> > > > > > > > fully
> > > >> > > > > > > > > > > > > available.
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > If we want to be on the safe side
> > with
> > > >> the
> > > >> > > > > > > > contribution,
> > > >> > > > > > > > > we
> > > >> > > > > > > > > > > > > should
> > > >> > > > > > > > > > > > > > > > > > contribute the source sooner and
> > > adjust
> > > >> it
> > > >> > > > later.
> > > >> > > > > > > That
> > > >> > > > > > > > > > would
> > > >> > > > > > > > > > > > also
> > > >> > > > > > > > > > > > > > > help
> > > >> > > > > > > > > > > > > > > > us
> > > >> > > > > > > > > > > > > > > > > > in case things get crazy towards
> the
> > > 1.10
> > > >> > > > feature
> > > >> > > > > > > > freeze
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > it
> > > >> > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > hard to find time to review the
> new
> > > >> > changes.
> > > >> > > > > > > > > > > > > > > > > > What would be the overhead of
> > > >> contributing
> > > >> > > now?
> > > >> > > > > > Given
> > > >> > > > > > > > > that
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > code
> > > >> > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > already there, it looks like it
> > would
> > > be
> > > >> > only
> > > >> > > > > > review
> > > >> > > > > > > > > > > capacity,
> > > >> > > > > > > > > > > > > right?
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > > > > > Stephan
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM
> > Yijie
> > > >> > Shen <
> > > >> > > > > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Hi everyone!
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Thanks for your attention and
> the
> > > >> > promotion
> > > >> > > > of
> > > >> > > > > > this
> > > >> > > > > > > > > work.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon
> as
> > > >> > possible
> > > >> > > > for
> > > >> > > > > > more
> > > >> > > > > > > > > > > specific
> > > >> > > > > > > > > > > > > > > > > discussions.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we
> have
> > > not
> > > >> > > > reached
> > > >> > > > > a
> > > >> > > > > > > > > > consensus.
> > > >> > > > > > > > > > > > > > > > Therefore,
> > > >> > > > > > > > > > > > > > > > > > > I will explain all the
> > > functionalities
> > > >> of
> > > >> > > the
> > > >> > > > > > > > existing
> > > >> > > > > > > > > > > > > connector in
> > > >> > > > > > > > > > > > > > > > > > > the FLIP (including Source,
> Sink,
> > > and
> > > >> > > > Catalog)
> > > >> > > > > to
> > > >> > > > > > > > > > continue
> > > >> > > > > > > > > > > > our
> > > >> > > > > > > > > > > > > > > > > > > discussions in FLIP.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Thanks for your kind help.
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > > > > > > Yijie
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM
> > > Becket
> > > >> > Qin
> > > >> > > <
> > > >> > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Hi Sijie,
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is
> to
> > > have
> > > >> > > Pulsar
> > > >> > > > > > > > connector
> > > >> > > > > > > > > > in
> > > >> > > > > > > > > > > > > 1.10,
> > > >> > > > > > > > > > > > > > > how
> > > >> > > > > > > > > > > > > > > > > > about
> > > >> > > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > do the following:
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar
> > > >> connector
> > > >> > > to
> > > >> > > > > > Flink
> > > >> > > > > > > > main
> > > >> > > > > > > > > > > repo
> > > >> > > > > > > > > > > > > as it
> > > >> > > > > > > > > > > > > > > > is a
> > > >> > > > > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > > public interface to Flink main
> > > repo.
> > > >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar
> > sink
> > > >> > right
> > > >> > > > away
> > > >> > > > > > as
> > > >> > > > > > > > > there
> > > >> > > > > > > > > > is
> > > >> > > > > > > > > > > > no
> > > >> > > > > > > > > > > > > > > change
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > sink interface so far.
> > > >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on
> FLIP-27.
> > > >> Flink
> > > >> > > 1.10
> > > >> > > > > is
> > > >> > > > > > > > going
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > be
> > > >> > > > > > > > > > > > > code
> > > >> > > > > > > > > > > > > > > > > freeze
> > > >> > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we
> give a
> > > >> month
> > > >> > to
> > > >> > > > the
> > > >> > > > > > > > > > development
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > review
> > > >> > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to
> > have
> > > >> > FLIP-27
> > > >> > > > by
> > > >> > > > > > late
> > > >> > > > > > > > > Oct.
> > > >> > > > > > > > > > > > There
> > > >> > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > still
> > > >> > > > > > > > > > > > > > > > > > 7
> > > >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it
> is
> > > >> doable.
> > > >> > > If
> > > >> > > > > > > FLIP-27
> > > >> > > > > > > > is
> > > >> > > > > > > > > > not
> > > >> > > > > > > > > > > > > ready
> > > >> > > > > > > > > > > > > > > by
> > > >> > > > > > > > > > > > > > > > > late
> > > >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check
> in
> > > >> Pulsar
> > > >> > > > > > connector
> > > >> > > > > > > > with
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > existing
> > > >> > > > > > > > > > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > > > > > > interface. This means we will
> > have
> > > >> > Pulsar
> > > >> > > > > > > connector
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > 1.10,
> > > >> > > > > > > > > > > > > > > > > > either
> > > >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Because we are going to have
> > > Pulsar
> > > >> > sink
> > > >> > > > and
> > > >> > > > > > > source
> > > >> > > > > > > > > > > checked
> > > >> > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > separately,
> > > >> > > > > > > > > > > > > > > > > > > > it might make sense to have
> two
> > > >> FLIPs,
> > > >> > > one
> > > >> > > > > for
> > > >> > > > > > > > Pulsar
> > > >> > > > > > > > > > > sink
> > > >> > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > another
> > > >> > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can
> start
> > > the
> > > >> > work
> > > >> > > on
> > > >> > > > > > > Pulsar
> > > >> > > > > > > > > sink
> > > >> > > > > > > > > > > > right
> > > >> > > > > > > > > > > > > > > away.
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM
> > > Sijie
> > > >> > Guo
> > > >> > > <
> > > >> > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink
> > > >> community?
> > > >> > > > Shall
> > > >> > > > > > we
> > > >> > > > > > > > wait
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > > > > FLIP-27
> > > >> > > > > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > > > > > > shall we
> > > >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And
> > what
> > > the
> > > >> > > next
> > > >> > > > > > steps
> > > >> > > > > > > > are?
> > > >> > > > > > > > > > :-)
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > > Sijie
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43
> PM
> > > >> Bowen
> > > >> > > Li <
> > > >> > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Hi,
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar
> > > connector
> > > >> > in
> > > >> > > > > Flink
> > > >> > > > > > > can
> > > >> > > > > > > > > be a
> > > >> > > > > > > > > > > > good
> > > >> > > > > > > > > > > > > > > mutual
> > > >> > > > > > > > > > > > > > > > > > > benefit
> > > >> > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > both communities.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Another perspective is
> that
> > > >> Pulsar
> > > >> > > > > > connector
> > > >> > > > > > > is
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > 1st
> > > >> > > > > > > > > > > > > > > > streaming
> > > >> > > > > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > > > > that integrates with
> Flink's
> > > >> > metadata
> > > >> > > > > > > > management
> > > >> > > > > > > > > > > system
> > > >> > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > Catalog
> > > >> > > > > > > > > > > > > > > > > > > APIs.
> > > >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how
> the
> > > >> > > > integration
> > > >> > > > > > > turns
> > > >> > > > > > > > > out
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > whether
> > > >> > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > need to
> > > >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog
> stack,
> > > >> which
> > > >> > > are
> > > >> > > > > > > > currently
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > > > Beta, to
> > > >> > > > > > > > > > > > > > > > > cater
> > > >> > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink.
> Thus
> > > I'm
> > > >> in
> > > >> > > > favor
> > > >> > > > > of
> > > >> > > > > > > > > merging
> > > >> > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > into
> > > >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit
> > smaller
> > > >> sized
> > > >> > > > PRs,
> > > >> > > > > > e.g.
> > > >> > > > > > > > > maybe
> > > >> > > > > > > > > > > one
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > basic
> > > >> > > > > > > > > > > > > > > > > > > > > > source/sink
> functionalities
> > > and
> > > >> > > another
> > > >> > > > > for
> > > >> > > > > > > > > schema
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > catalog
> > > >> > > > > > > > > > > > > > > > > > > > > integration,
> > > >> > > > > > > > > > > > > > > > > > > > > > just to make them easier
> to
> > > >> review.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to
> > > wait
> > > >> for
> > > >> > > > > > FLIP-27.
> > > >> > > > > > > > But
> > > >> > > > > > > > > I
> > > >> > > > > > > > > > > > don't
> > > >> > > > > > > > > > > > > > > think
> > > >> > > > > > > > > > > > > > > > > > > FLIP-27
> > > >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in
> cases
> > > >> where
> > > >> > it
> > > >> > > > > > cannot
> > > >> > > > > > > > make
> > > >> > > > > > > > > > its
> > > >> > > > > > > > > > > > way
> > > >> > > > > > > > > > > > > > > into
> > > >> > > > > > > > > > > > > > > > > 1.10
> > > >> > > > > > > > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable
> > > amount
> > > >> of
> > > >> > > time
> > > >> > > > > for
> > > >> > > > > > > > > > > committers
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > review
> > > >> > > > > > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully
> > > adapt
> > > >> to
> > > >> > > new
> > > >> > > > > > > > > interfaces.
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > Bowen
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> 3:21
> > AM
> > > >> > Becket
> > > >> > > > > Qin <
> > > >> > > > > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Till,
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > You are right. It all
> > > depends
> > > >> on
> > > >> > > when
> > > >> > > > > the
> > > >> > > > > > > new
> > > >> > > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > > interface
> > > >> > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > going
> > > >> > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I
> > > think it
> > > >> > > would
> > > >> > > > > be
> > > >> > > > > > > > there
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > > > about a
> > > >> > > > > > > > > > > > > > > > month
> > > >> > > > > > > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > > > > > > so.
> > > >> > > > > > > > > > > > > > > > > > > > > > But
> > > >> > > > > > > > > > > > > > > > > > > > > > > I could be too
> optimistic.
> > > It
> > > >> > would
> > > >> > > > > also
> > > >> > > > > > be
> > > >> > > > > > > > > good
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > > hear
> > > >> > > > > > > > > > > > > > > what
> > > >> > > > > > > > > > > > > > > > > do
> > > >> > > > > > > > > > > > > > > > > > > > > Aljoscha
> > > >> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as
> they
> > > are
> > > >> > also
> > > >> > > > > > involved
> > > >> > > > > > > > in
> > > >> > > > > > > > > > > > FLIP-27.
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > In general I think we
> > should
> > > >> have
> > > >> > > > > Pulsar
> > > >> > > > > > > > > > connector
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > 1.10,
> > > >> > > > > > > > > > > > > > > > > > > > > > > preferably with the new
> > > source
> > > >> > > > > interface.
> > > >> > > > > > > We
> > > >> > > > > > > > > can
> > > >> > > > > > > > > > > also
> > > >> > > > > > > > > > > > > check
> > > >> > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > right
> > > >> > > > > > > > > > > > > > > > > > > > > > now
> > > >> > > > > > > > > > > > > > > > > > > > > > > with old source
> interface,
> > > but
> > > >> I
> > > >> > > > > suspect
> > > >> > > > > > > few
> > > >> > > > > > > > > > users
> > > >> > > > > > > > > > > > > will use
> > > >> > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > before
> > > >> > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > next official release.
> > > >> Therefore,
> > > >> > > it
> > > >> > > > > > seems
> > > >> > > > > > > > > > > reasonable
> > > >> > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > wait a
> > > >> > > > > > > > > > > > > > > > > > > little
> > > >> > > > > > > > > > > > > > > > > > > > > > bit
> > > >> > > > > > > > > > > > > > > > > > > > > > > to see whether we can
> jump
> > > to
> > > >> the
> > > >> > > new
> > > >> > > > > > > source
> > > >> > > > > > > > > > > > > interface. As
> > > >> > > > > > > > > > > > > > > > long
> > > >> > > > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > > > make
> > > >> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it,
> > > >> waiting a
> > > >> > > > > little
> > > >> > > > > > > bit
> > > >> > > > > > > > > > > doesn't
> > > >> > > > > > > > > > > > > seem
> > > >> > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > hurt
> > > >> > > > > > > > > > > > > > > > > > > much.
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> > 3:59
> > > PM
> > > >> > Till
> > > >> > > > > > > Rohrmann
> > > >> > > > > > > > <
> > > >> > > > > > > > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone,
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the
> > > >> problem
> > > >> > > > would
> > > >> > > > > be
> > > >> > > > > > > if
> > > >> > > > > > > > we
> > > >> > > > > > > > > > > > > committed
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > connector before the
> new
> > > >> source
> > > >> > > > > > interface
> > > >> > > > > > > > is
> > > >> > > > > > > > > > > ready.
> > > >> > > > > > > > > > > > > If I
> > > >> > > > > > > > > > > > > > > > > > > understood
> > > >> > > > > > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we
> need
> > to
> > > >> > > support
> > > >> > > > > the
> > > >> > > > > > > old
> > > >> > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > interface
> > > >> > > > > > > > > > > > > > > > > > > anyway
> > > >> > > > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > existing connectors.
> By
> > > >> > checking
> > > >> > > it
> > > >> > > > > in
> > > >> > > > > > > > early
> > > >> > > > > > > > > I
> > > >> > > > > > > > > > > > could
> > > >> > > > > > > > > > > > > see
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > benefit
> > > >> > > > > > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > > > > > our users could start
> > > using
> > > >> the
> > > >> > > > > > connector
> > > >> > > > > > > > > > > earlier.
> > > >> > > > > > > > > > > > > > > > Moreover,
> > > >> > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > > > > > > > > > prevent that the
> Pulsar
> > > >> > > integration
> > > >> > > > > is
> > > >> > > > > > > > being
> > > >> > > > > > > > > > > > delayed
> > > >> > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > case
> > > >> > > > > > > > > > > > > > > > > > > that the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > source interface
> should
> > be
> > > >> > > delayed.
> > > >> > > > > The
> > > >> > > > > > > > only
> > > >> > > > > > > > > > > > > downside I
> > > >> > > > > > > > > > > > > > > see
> > > >> > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > extra
> > > >> > > > > > > > > > > > > > > > > > > > > > > > review effort and
> > > potential
> > > >> > fixes
> > > >> > > > > which
> > > >> > > > > > > > might
> > > >> > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > irrelevant
> > > >> > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > > > > > > source interface
> > > >> > implementation.
> > > >> > > I
> > > >> > > > > > guess
> > > >> > > > > > > it
> > > >> > > > > > > > > > > mainly
> > > >> > > > > > > > > > > > > > > depends
> > > >> > > > > > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > > > > > > how
> > > >> > > > > > > > > > > > > > > > > > > > > > certain
> > > >> > > > > > > > > > > > > > > > > > > > > > > > we are when the new
> > source
> > > >> > > > interface
> > > >> > > > > > will
> > > >> > > > > > > > be
> > > >> > > > > > > > > > > ready.
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Till
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
> > > 8:56
> > > >> AM
> > > >> > > > Becket
> > > >> > > > > > > Qin <
> > > >> > > > > > > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing
> > your
> > > >> > > thoughts.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have
> some
> > > >> update
> > > >> > > on
> > > >> > > > > > > FLIP-27.
> > > >> > > > > > > > > > > > Although
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > FLIP
> > > >> > > > > > > > > > > > > > > > > > > wiki
> > > >> > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread
> has
> > > been
> > > >> > > quiet
> > > >> > > > > for
> > > >> > > > > > > some
> > > >> > > > > > > > > > > time, a
> > > >> > > > > > > > > > > > > few
> > > >> > > > > > > > > > > > > > > > > > > committer /
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > contributors in
> Flink
> > > >> > community
> > > >> > > > > were
> > > >> > > > > > > > > actually
> > > >> > > > > > > > > > > > > > > prototyping
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > entire
> > > >> > > > > > > > > > > > > > > > > > > > > > > > thing.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > We have made some
> good
> > > >> > progress
> > > >> > > > > there
> > > >> > > > > > > but
> > > >> > > > > > > > > > want
> > > >> > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > update
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > FLIP
> > > >> > > > > > > > > > > > > > > > > > > > > > wiki
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > after the entire
> thing
> > > is
> > > >> > > > verified
> > > >> > > > > to
> > > >> > > > > > > > work
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > > case
> > > >> > > > > > > > > > > > > > > there
> > > >> > > > > > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > > some
> > > >> > > > > > > > > > > > > > > > > > > > > > last
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in
> the
> > > >> > > > > > implementation.
> > > >> > > > > > > I
> > > >> > > > > > > > > > don't
> > > >> > > > > > > > > > > > > have an
> > > >> > > > > > > > > > > > > > > > > exact
> > > >> > > > > > > > > > > > > > > > > > > ETA
> > > >> > > > > > > > > > > > > > > > > > > > > yet,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > but I
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to
> > be
> > > >> > within
> > > >> > > a
> > > >> > > > > > month
> > > >> > > > > > > or
> > > >> > > > > > > > > so.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review
> > the
> > > >> > > current
> > > >> > > > > > Flink
> > > >> > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > see if
> > > >> > > > > > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It
> > > would be
> > > >> > > good
> > > >> > > > to
> > > >> > > > > > > avoid
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > case
> > > >> > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > checked
> > > >> > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector
> with
> > > some
> > > >> > > review
> > > >> > > > > > > efforts
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > > > shortly
> > > >> > > > > > > > > > > > > > > > after
> > > >> > > > > > > > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is
> > > ready.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket)
> Qin
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019
> at
> > > 8:39
> > > >> > AM
> > > >> > > > > Yijie
> > > >> > > > > > > > Shen <
> > > >> > > > > > > > > > > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the
> > > >> feedback
> > > >> > > and
> > > >> > > > > > > > > > suggestions!
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the
> > > goal
> > > >> of
> > > >> > > the
> > > >> > > > > > > > connector
> > > >> > > > > > > > > > has
> > > >> > > > > > > > > > > > > always
> > > >> > > > > > > > > > > > > > > > been
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > provide
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > users with the
> > latest
> > > >> > > features
> > > >> > > > of
> > > >> > > > > > > both
> > > >> > > > > > > > > > > systems
> > > >> > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > soon
> > > >> > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > > > possible.
> > > >> > > > > > > > > > > > > > > > > > > > > > > We
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > propose to
> > contribute
> > > the
> > > >> > > > > connector
> > > >> > > > > > > to
> > > >> > > > > > > > > > Flink
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > hope
> > > >> > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > get
> > > >> > > > > > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and
> > > feedback
> > > >> > from
> > > >> > > > > Flink
> > > >> > > > > > > > > experts
> > > >> > > > > > > > > > > to
> > > >> > > > > > > > > > > > > ensure
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > high
> > > >> > > > > > > > > > > > > > > > > > > > > > > quality
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we
> > > noticed
> > > >> its
> > > >> > > > > > existence
> > > >> > > > > > > > at
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > beginning
> > > >> > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > reworking
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the connector
> > > >> > implementation
> > > >> > > > > based
> > > >> > > > > > on
> > > >> > > > > > > > > Flink
> > > >> > > > > > > > > > > > 1.9;
> > > >> > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > also
> > > >> > > > > > > > > > > > > > > > > > > wanted
> > > >> > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector
> > that
> > > >> > > supports
> > > >> > > > > > both
> > > >> > > > > > > > > batch
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > > stream
> > > >> > > > > > > > > > > > > > > > > > > computing
> > > >> > > > > > > > > > > > > > > > > > > > > > based
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > on it.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has
> been
> > > >> > inactive
> > > >> > > > for
> > > >> > > > > > > some
> > > >> > > > > > > > > > time,
> > > >> > > > > > > > > > > so
> > > >> > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > decided
> > > >> > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > provide
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with
> > most
> > > of
> > > >> > the
> > > >> > > > new
> > > >> > > > > > > > > features,
> > > >> > > > > > > > > > > such
> > > >> > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > type
> > > >> > > > > > > > > > > > > > > > > > > > > > > system
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > and the new
> catalog
> > > API
> > > >> > > first.
> > > >> > > > We
> > > >> > > > > > > will
> > > >> > > > > > > > > pay
> > > >> > > > > > > > > > > > > attention
> > > >> > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > progress
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27
> > continually
> > > >> and
> > > >> > > > > > > incorporate
> > > >> > > > > > > > it
> > > >> > > > > > > > > > > with
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > > > > soon
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > as possible.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test
> > > status
> > > >> > of
> > > >> > > > the
> > > >> > > > > > > > > connector,
> > > >> > > > > > > > > > > we
> > > >> > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > following
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > other
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test
> in
> > > Flink
> > > >> > > > > > repository
> > > >> > > > > > > > and
> > > >> > > > > > > > > > > aimed
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > provide
> > > >> > > > > > > > > > > > > > > > > > > > > > throughout
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could.
> > We
> > > are
> > > >> > > also
> > > >> > > > > > happy
> > > >> > > > > > > to
> > > >> > > > > > > > > > hear
> > > >> > > > > > > > > > > > > > > > suggestions
> > > >> > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from
> the
> > > >> Flink
> > > >> > > > > > community
> > > >> > > > > > > to
> > > >> > > > > > > > > > > improve
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > stability and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the
> > > >> > connector
> > > >> > > > > > > > > continuously.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Best,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5,
> 2019
> > at
> > > >> 5:59
> > > >> > > AM
> > > >> > > > > > Sijie
> > > >> > > > > > > > Guo
> > > >> > > > > > > > > <
> > > >> > > > > > > > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone
> > for
> > > the
> > > >> > > > > comments
> > > >> > > > > > > and
> > > >> > > > > > > > > > > > feedback.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me
> > that
> > > the
> > > >> > > main
> > > >> > > > > > > question
> > > >> > > > > > > > > > here
> > > >> > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > about -
> > > >> > > > > > > > > > > > > > > > > > "how
> > > >> > > > > > > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community
> maintain
> > > the
> > > >> > > > > > connector?".
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two
> > > thoughts
> > > >> > from
> > > >> > > > > > myself.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how
> and
> > > >> where
> > > >> > to
> > > >> > > > > host
> > > >> > > > > > > this
> > > >> > > > > > > > > > > > > integration
> > > >> > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > kind
> > > >> > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > less
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > important
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe
> > > there
> > > >> can
> > > >> > > be
> > > >> > > > > many
> > > >> > > > > > > > ways
> > > >> > > > > > > > > to
> > > >> > > > > > > > > > > > > achieve
> > > >> > > > > > > > > > > > > > > it.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the
> > > >> > > contribution,
> > > >> > > > > what
> > > >> > > > > > > we
> > > >> > > > > > > > > are
> > > >> > > > > > > > > > > > > looking
> > > >> > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > here
> > > >> > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > how
> > > >> > > > > > > > > > > > > > > > > > > > > > > > these
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > two
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can
> > > build
> > > >> the
> > > >> > > > > > > > collaboration
> > > >> > > > > > > > > > > > > > > relationship
> > > >> > > > > > > > > > > > > > > > on
> > > >> > > > > > > > > > > > > > > > > > > > > > developing
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration
> > > between
> > > >> > > > Pulsar
> > > >> > > > > > and
> > > >> > > > > > > > > Flink.
> > > >> > > > > > > > > > > > Even
> > > >> > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > try
> > > >> > > > > > > > > > > > > > > > > > > our
> > > >> > > > > > > > > > > > > > > > > > > > > > best
> > > >> > > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > catch
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the
> updates
> > > in
> > > >> > Flink
> > > >> > > > > > > > community.
> > > >> > > > > > > > > We
> > > >> > > > > > > > > > > are
> > > >> > > > > > > > > > > > > still
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact
> > > that we
> > > >> > > have
> > > >> > > > > less
> > > >> > > > > > > > > > > experiences
> > > >> > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > than
> > > >> > > > > > > > > > > > > > > > > > > > > folks
> > > >> > > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In
> > order
> > > to
> > > >> > make
> > > >> > > > > sure
> > > >> > > > > > we
> > > >> > > > > > > > > > > maintain
> > > >> > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > deliver
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality
> > > >> > pulsar-flink
> > > >> > > > > > > > integration
> > > >> > > > > > > > > > to
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > users
> > > >> > > > > > > > > > > > > > > > > who
> > > >> > > > > > > > > > > > > > > > > > > use
> > > >> > > > > > > > > > > > > > > > > > > > > both
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we
> > > need
> > > >> > some
> > > >> > > > help
> > > >> > > > > > > from
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > > experts
> > > >> > > > > > > > > > > > > > > from
> > > >> > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > community.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been
> > > >> following
> > > >> > > > > FLIP-27
> > > >> > > > > > > > for a
> > > >> > > > > > > > > > > > while.
> > > >> > > > > > > > > > > > > > > > > Originally
> > > >> > > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > > were
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > thinking
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing
> > the
> > > >> > > > connectors
> > > >> > > > > > back
> > > >> > > > > > > > > after
> > > >> > > > > > > > > > > > > > > integrating
> > > >> > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > new API
> introduced
> > > in
> > > >> > > > FLIP-27.
> > > >> > > > > > But
> > > >> > > > > > > we
> > > >> > > > > > > > > > > decided
> > > >> > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > initiate
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > conversation
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as
> > > possible.
> > > >> > > Because
> > > >> > > > > we
> > > >> > > > > > > > > believe
> > > >> > > > > > > > > > > > there
> > > >> > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > > > > benefits
> > > >> > > > > > > > > > > > > > > > > > > > > > > > doing
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather
> than
> > > >> later.
> > > >> > > As
> > > >> > > > > part
> > > >> > > > > > > of
> > > >> > > > > > > > > > > > > contribution,
> > > >> > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > > > help
> > > >> > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community
> > understand
> > > >> more
> > > >> > > > about
> > > >> > > > > > > > Pulsar
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > potential
> > > >> > > > > > > > > > > > > > > > > > > > > > > integration
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > points.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also
> > > help
> > > >> > Flink
> > > >> > > > > > > community
> > > >> > > > > > > > > > > verify
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > well
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API
> > > (e.g.
> > > >> > > > catalog
> > > >> > > > > > > API).
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> > 2019
> > > at
> > > >> > 5:24
> > > >> > > > AM
> > > >> > > > > > > Becket
> > > >> > > > > > > > > > Qin <
> > > >> > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the
> > > >> interest
> > > >> > > in
> > > >> > > > > > > > > contributing
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > connector.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I
> > > think
> > > >> > > having
> > > >> > > > > > Pulsar
> > > >> > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > strong
> > > >> > > > > > > > > > > > > > > > > > > > > support
> > > >> > > > > > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable
> > addition
> > > to
> > > >> > > Flink.
> > > >> > > > > So
> > > >> > > > > > I
> > > >> > > > > > > am
> > > >> > > > > > > > > > happy
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > shepherd
> > > >> > > > > > > > > > > > > > > > > > > this
> > > >> > > > > > > > > > > > > > > > > > > > > > > effort.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I
> > would
> > > >> also
> > > >> > > > like
> > > >> > > > > to
> > > >> > > > > > > > > provide
> > > >> > > > > > > > > > > > some
> > > >> > > > > > > > > > > > > > > > context
> > > >> > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > recent
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink
> > > connectors
> > > >> > > > > ecosystem.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current
> way
> > > Flink
> > > >> > > > > maintains
> > > >> > > > > > > its
> > > >> > > > > > > > > > > > > connector has
> > > >> > > > > > > > > > > > > > > > hit
> > > >> > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > scalability
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > bar.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and
> > more
> > > >> > > > connectors
> > > >> > > > > > > > coming
> > > >> > > > > > > > > > into
> > > >> > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > repo,
> > > >> > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > > > > > > > facing a
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > few
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such
> as
> > > long
> > > >> > > build
> > > >> > > > > and
> > > >> > > > > > > > > testing
> > > >> > > > > > > > > > > > > time. To
> > > >> > > > > > > > > > > > > > > > > > address
> > > >> > > > > > > > > > > > > > > > > > > this
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > problem,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted
> > to
> > > do
> > > >> > the
> > > >> > > > > > > following:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out
> the
> > > >> > > connectors
> > > >> > > > > > into
> > > >> > > > > > > a
> > > >> > > > > > > > > > > separate
> > > >> > > > > > > > > > > > > > > > > repository.
> > > >> > > > > > > > > > > > > > > > > > > This
> > > >> > > > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to
> > > >> > potential
> > > >> > > > > > solution
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > > shorten
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > build
> > > >> > > > > > > > > > > > > > > > > > > time.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage
> the
> > > >> > > connectors
> > > >> > > > > to
> > > >> > > > > > > stay
> > > >> > > > > > > > > as
> > > >> > > > > > > > > > > > > ecosystem
> > > >> > > > > > > > > > > > > > > > > > project
> > > >> > > > > > > > > > > > > > > > > > > > > while
> > > >> > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > tries
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide
> good
> > > >> support
> > > >> > > for
> > > >> > > > > > > > > > functionality
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > compatibility
> > > >> > > > > > > > > > > > > > > > > > > > > > > tests.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Robert
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to
> > > create
> > > >> a
> > > >> > > > Flink
> > > >> > > > > > > > > Ecosystem
> > > >> > > > > > > > > > > > > project
> > > >> > > > > > > > > > > > > > > > > website
> > > >> > > > > > > > > > > > > > > > > > > and it
> > > >> > > > > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > going
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some
> > final
> > > >> > > approval
> > > >> > > > > > > > process.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the
> above
> > > >> > efforts,
> > > >> > > it
> > > >> > > > > > would
> > > >> > > > > > > > be
> > > >> > > > > > > > > > > great
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > first
> > > >> > > > > > > > > > > > > > > > > see
> > > >> > > > > > > > > > > > > > > > > > > if we
> > > >> > > > > > > > > > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > have
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> connector
> > > as
> > > >> an
> > > >> > > > > > ecosystem
> > > >> > > > > > > > > > project
> > > >> > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > great
> > > >> > > > > > > > > > > > > > > > > > > support.
> > > >> > > > > > > > > > > > > > > > > > > > > It
> > > >> > > > > > > > > > > > > > > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear
> how
> > > the
> > > >> > > Flink
> > > >> > > > > > Pulsar
> > > >> > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > tested
> > > >> > > > > > > > > > > > > > > > > > > > > currently
> > > >> > > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > see
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > if
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn
> > > >> something
> > > >> > to
> > > >> > > > > > > maintain
> > > >> > > > > > > > it
> > > >> > > > > > > > > > as
> > > >> > > > > > > > > > > an
> > > >> > > > > > > > > > > > > > > > ecosystem
> > > >> > > > > > > > > > > > > > > > > > > project
> > > >> > > > > > > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > good
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and
> test
> > > >> > > coverage.
> > > >> > > > If
> > > >> > > > > > the
> > > >> > > > > > > > > > quality
> > > >> > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > an
> > > >> > > > > > > > > > > > > > > > > > ecosystem
> > > >> > > > > > > > > > > > > > > > > > > > > > project
> > > >> > > > > > > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > hard
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee,
> we
> > > may
> > > >> as
> > > >> > > > well
> > > >> > > > > > > adopt
> > > >> > > > > > > > it
> > > >> > > > > > > > > > > into
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > main
> > > >> > > > > > > > > > > > > > > > > > repo.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another
> > > ongoing
> > > >> > > effort
> > > >> > > > > is
> > > >> > > > > > > > > FLIP-27
> > > >> > > > > > > > > > > > where
> > > >> > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > > making
> > > >> > > > > > > > > > > > > > > > > > > > > > > changes
> > > >> > > > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source
> > > >> connector
> > > >> > > > > > > architecture
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > > > > > interface.
> > > >> > > > > > > > > > > > > > > > > This
> > > >> > > > > > > > > > > > > > > > > > > > > change
> > > >> > > > > > > > > > > > > > > > > > > > > > > will
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > likely
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10.
> > > >> Therefore
> > > >> > > > > timing
> > > >> > > > > > > > wise,
> > > >> > > > > > > > > if
> > > >> > > > > > > > > > > we
> > > >> > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > going
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > have
> > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in
> > main
> > > >> > repo, I
> > > >> > > > am
> > > >> > > > > > > > > wondering
> > > >> > > > > > > > > > if
> > > >> > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > should
> > > >> > > > > > > > > > > > > > > > > > hold
> > > >> > > > > > > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > > > > > little
> > > >> > > > > > > > > > > > > > > > > > > > > > > > bit
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar
> > > >> > connector
> > > >> > > > > adapt
> > > >> > > > > > to
> > > >> > > > > > > > the
> > > >> > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > interface
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > avoid
> > > >> > > > > > > > > > > > > > > > > > > > > > > > shortly
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated
> work?
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie
> > (Becket)
> > > Qin
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4,
> > > 2019
> > > >> at
> > > >> > > 4:32
> > > >> > > > > PM
> > > >> > > > > > > > > Chesnay
> > > >> > > > > > > > > > > > > Schepler <
> > > >> > > > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite
> > > worried
> > > >> > that
> > > >> > > we
> > > >> > > > > may
> > > >> > > > > > > end
> > > >> > > > > > > > > up
> > > >> > > > > > > > > > > > > repeating
> > > >> > > > > > > > > > > > > > > > > > history.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were
> > > already
> > > >> 2
> > > >> > > > > attempts
> > > >> > > > > > > at
> > > >> > > > > > > > > > > > > contributing a
> > > >> > > > > > > > > > > > > > > > > > pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > connector,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > both
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which
> > failed
> > > >> > because
> > > >> > > > no
> > > >> > > > > > > > > committer
> > > >> > > > > > > > > > > was
> > > >> > > > > > > > > > > > > > > getting
> > > >> > > > > > > > > > > > > > > > > > > involved,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > despite
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor
> > > >> opening a
> > > >> > > > > > dedicated
> > > >> > > > > > > > > > > > discussion
> > > >> > > > > > > > > > > > > > > thread
> > > >> > > > > > > > > > > > > > > > > > > about the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > contribution
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand
> and
> > > >> > getting
> > > >> > > > > > several
> > > >> > > > > > > > +1's
> > > >> > > > > > > > > > > from
> > > >> > > > > > > > > > > > > > > > > committers.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should
> > really
> > > >> make
> > > >> > > > sure
> > > >> > > > > > that
> > > >> > > > > > > > if
> > > >> > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > welcome/approve
> > > >> > > > > > > > > > > > > > > > > > > such
> > > >> > > > > > > > > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution
> > it
> > > >> will
> > > >> > > > > actually
> > > >> > > > > > > get
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > > > > attention
> > > >> > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > > > deserves.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm
> > > >> inclined
> > > >> > > to
> > > >> > > > > > > > recommend
> > > >> > > > > > > > > > > > > maintaining
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > outside
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We
> > could
> > > >> link
> > > >> > to
> > > >> > > > it
> > > >> > > > > > from
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > > > documentation
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > give
> > > >> > > > > > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the
> > > upcoming
> > > >> > page
> > > >> > > > for
> > > >> > > > > > > > sharing
> > > >> > > > > > > > > > > > > artifacts
> > > >> > > > > > > > > > > > > > > > among
> > > >> > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > community
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > (what's
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of
> > > that
> > > >> > > > anyway?),
> > > >> > > > > > > this
> > > >> > > > > > > > > may
> > > >> > > > > > > > > > > be a
> > > >> > > > > > > > > > > > > > > better
> > > >> > > > > > > > > > > > > > > > > > > option.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On
> 04/09/2019
> > > >> 10:16,
> > > >> > > Till
> > > >> > > > > > > > Rohrmann
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi
> everyone,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a
> lot
> > > for
> > > >> > > > starting
> > > >> > > > > > > this
> > > >> > > > > > > > > > > > discussion
> > > >> > > > > > > > > > > > > > > > Yijie.
> > > >> > > > > > > > > > > > > > > > > I
> > > >> > > > > > > > > > > > > > > > > > > think
> > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector
> > > would
> > > >> be
> > > >> > a
> > > >> > > > very
> > > >> > > > > > > > > valuable
> > > >> > > > > > > > > > > > > addition
> > > >> > > > > > > > > > > > > > > > since
> > > >> > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > becomes
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more
> popular
> > > and
> > > >> it
> > > >> > > > would
> > > >> > > > > > > > further
> > > >> > > > > > > > > > > > expand
> > > >> > > > > > > > > > > > > > > > Flink's
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > interoperability.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Also
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a
> > project
> > > >> > > > > perspective
> > > >> > > > > > it
> > > >> > > > > > > > > makes
> > > >> > > > > > > > > > > > > sense for
> > > >> > > > > > > > > > > > > > > > me
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > place
> > > >> > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > downstream
> > > >> > > project.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main
> > > >> > > > concern/question
> > > >> > > > > is
> > > >> > > > > > > how
> > > >> > > > > > > > > can
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > community
> > > >> > > > > > > > > > > > > > > > > > > > > > > > maintain
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector?
> > We
> > > >> have
> > > >> > > seen
> > > >> > > > > in
> > > >> > > > > > > the
> > > >> > > > > > > > > past
> > > >> > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > > > connectors
> > > >> > > > > > > > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > > > > > some
> > > >> > > > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > most
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively
> > > >> developed
> > > >> > > > > > components
> > > >> > > > > > > > > > because
> > > >> > > > > > > > > > > > > they
> > > >> > > > > > > > > > > > > > > need
> > > >> > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > be
> > > >> > > > > > > > > > > > > > > > > > > > > kept
> > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > sync
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external
> > > system
> > > >> and
> > > >> > > > with
> > > >> > > > > > > Flink.
> > > >> > > > > > > > > > Given
> > > >> > > > > > > > > > > > > that
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > community
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help
> with
> > > >> > > > maintaining,
> > > >> > > > > > > > > improving
> > > >> > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > evolving
> > > >> > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > connector,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > I'm
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic
> > > that
> > > >> we
> > > >> > > can
> > > >> > > > > > > achieve
> > > >> > > > > > > > > > this.
> > > >> > > > > > > > > > > > > Hence,
> > > >> > > > > > > > > > > > > > > +1
> > > >> > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > contributing
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > back
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed,
> Sep
> > 4,
> > > >> 2019
> > > >> > > at
> > > >> > > > > 2:03
> > > >> > > > > > > AM
> > > >> > > > > > > > > > Sijie
> > > >> > > > > > > > > > > > Guo
> > > >> > > > > > > > > > > > > <
> > > >> > > > > > > > > > > > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I
> was
> > > the
> > > >> > main
> > > >> > > > > > driver
> > > >> > > > > > > > > behind
> > > >> > > > > > > > > > > > > > > FLINK-9641
> > > >> > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > let
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > me
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more
> > > context
> > > >> > on
> > > >> > > > > this.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> FLINK-9641
> > > and
> > > >> > > > > FLINK-9168
> > > >> > > > > > > was
> > > >> > > > > > > > > > > created
> > > >> > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > bringing
> > > >> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for
> > > Flink.
> > > >> > The
> > > >> > > > > > > > integration
> > > >> > > > > > > > > > was
> > > >> > > > > > > > > > > > done
> > > >> > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > 1.6.0.
> > > >> > > > > > > > > > > > > > > > > > > > > > > We
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > sent out
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests
> > > about a
> > > >> > > year
> > > >> > > > > ago
> > > >> > > > > > > and
> > > >> > > > > > > > we
> > > >> > > > > > > > > > > ended
> > > >> > > > > > > > > > > > > up
> > > >> > > > > > > > > > > > > > > > > > > maintaining
> > > >> > > > > > > > > > > > > > > > > > > > > > those
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar
> for
> > > >> Pulsar
> > > >> > > > users
> > > >> > > > > to
> > > >> > > > > > > use
> > > >> > > > > > > > > > Flink
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > process
> > > >> > > > > > > > > > > > > > > > > > > event
> > > >> > > > > > > > > > > > > > > > > > > > > > > streams
> > > >> > > > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > ).
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > The
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6
> > > integration
> > > >> is
> > > >> > > > > pretty
> > > >> > > > > > > > simple
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > > > there
> > > >> > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > no
> > > >> > > > > > > > > > > > > > > > > > > schema
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> considerations.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the
> past
> > > >> year,
> > > >> > we
> > > >> > > > > have
> > > >> > > > > > > > made a
> > > >> > > > > > > > > > lot
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > changes
> > > >> > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > brought
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar
> > > schema as
> > > >> > the
> > > >> > > > > > > > first-class
> > > >> > > > > > > > > > > > > citizen in
> > > >> > > > > > > > > > > > > > > > > > Pulsar.
> > > >> > > > > > > > > > > > > > > > > > > We
> > > >> > > > > > > > > > > > > > > > > > > > > > also
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > integrated
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other
> > > computing
> > > >> > > > engines
> > > >> > > > > > for
> > > >> > > > > > > > > > > processing
> > > >> > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > event
> > > >> > > > > > > > > > > > > > > > > > > > > > streams
> > > >> > > > > > > > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us
> > to
> > > >> > rethink
> > > >> > > > how
> > > >> > > > > > to
> > > >> > > > > > > > > > > integrate
> > > >> > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > best
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > way.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Then
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> reimplement
> > > the
> > > >> > > > > > pulsar-flink
> > > >> > > > > > > > > > > > connectors
> > > >> > > > > > > > > > > > > from
> > > >> > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > ground
> > > >> > > > > > > > > > > > > > > > > > > > > up
> > > >> > > > > > > > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring
> table
> > > API
> > > >> > and
> > > >> > > > > > catalog
> > > >> > > > > > > > API
> > > >> > > > > > > > > as
> > > >> > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > first-class
> > > >> > > > > > > > > > > > > > > > > > > > > citizen
> > > >> > > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > integration.
> > > >> With
> > > >> > > that
> > > >> > > > > > being
> > > >> > > > > > > > > said,
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > > > pulsar-flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > implementation,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can
> > > register
> > > >> > > > pulsar
> > > >> > > > > > as a
> > > >> > > > > > > > > flink
> > > >> > > > > > > > > > > > > catalog
> > > >> > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > query /
> > > >> > > > > > > > > > > > > > > > > > > > > > > process
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > event
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams
> > using
> > > >> > Flink
> > > >> > > > SQL.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is
> an
> > > >> example
> > > >> > > > about
> > > >> > > > > > how
> > > >> > > > > > > > to
> > > >> > > > > > > > > > use
> > > >> > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > a
> > > >> > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > catalog:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has
> > > also
> > > >> > > > written a
> > > >> > > > > > > blog
> > > >> > > > > > > > > post
> > > >> > > > > > > > > > > > > > > explaining
> > > >> > > > > > > > > > > > > > > > > why
> > > >> > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > re-implement
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector
> > > with
> > > >> > Flink
> > > >> > > > 1.9
> > > >> > > > > > and
> > > >> > > > > > > > > what
> > > >> > > > > > > > > > > are
> > > >> > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > changes
> > > >> > > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > > > made
> > > >> > > > > > > > > > > > > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > new
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> connector:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We
> believe
> > > >> Pulsar
> > > >> > is
> > > >> > > > not
> > > >> > > > > > > just
> > > >> > > > > > > > a
> > > >> > > > > > > > > > > simple
> > > >> > > > > > > > > > > > > data
> > > >> > > > > > > > > > > > > > > > sink
> > > >> > > > > > > > > > > > > > > > > > or
> > > >> > > > > > > > > > > > > > > > > > > > > source
> > > >> > > > > > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > It
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually
> > can
> > > be
> > > >> a
> > > >> > > > fully
> > > >> > > > > > > > > integrated
> > > >> > > > > > > > > > > > > streaming
> > > >> > > > > > > > > > > > > > > > > data
> > > >> > > > > > > > > > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas
> > (sink,
> > > >> > source,
> > > >> > > > > > > > > > schema/catalog
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > state).
> > > >> > > > > > > > > > > > > > > > > > The
> > > >> > > > > > > > > > > > > > > > > > > > > > > > combination
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and
> Pulsar
> > > can
> > > >> > > create
> > > >> > > > a
> > > >> > > > > > > great
> > > >> > > > > > > > > > > > streaming
> > > >> > > > > > > > > > > > > > > > > warehouse
> > > >> > > > > > > > > > > > > > > > > > > > > > > architecture
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > for
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > streaming-first,
> > > >> > > > unified
> > > >> > > > > > > data
> > > >> > > > > > > > > > > > > processing.
> > > >> > > > > > > > > > > > > > > > Since
> > > >> > > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > > > > > > talking
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> contribute
> > > >> Pulsar
> > > >> > > > > > > integration
> > > >> > > > > > > > to
> > > >> > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > here,
> > > >> > > > > > > > > > > > > > > > we
> > > >> > > > > > > > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > > > > > > also
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain,
> > > >> improve
> > > >> > > and
> > > >> > > > > > evolve
> > > >> > > > > > > > the
> > > >> > > > > > > > > > > > > integration
> > > >> > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > help
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use
> > both
> > > >> Flink
> > > >> > > and
> > > >> > > > > > > Pulsar.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this
> > > give
> > > >> > you a
> > > >> > > > bit
> > > >> > > > > > > more
> > > >> > > > > > > > > > > > background
> > > >> > > > > > > > > > > > > > > about
> > > >> > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > integration.
> > > Let
> > > >> > me
> > > >> > > > know
> > > >> > > > > > > what
> > > >> > > > > > > > > are
> > > >> > > > > > > > > > > your
> > > >> > > > > > > > > > > > > > > > thoughts.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue,
> Sep
> > > 3,
> > > >> > 2019
> > > >> > > at
> > > >> > > > > > 11:54
> > > >> > > > > > > > AM
> > > >> > > > > > > > > > Yun
> > > >> > > > > > > > > > > > > Tang <
> > > >> > > > > > > > > > > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can
> see
> > > that
> > > >> > > Pulsar
> > > >> > > > > > > becomes
> > > >> > > > > > > > > > more
> > > >> > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > more
> > > >> > > > > > > > > > > > > > > > > > popular
> > > >> > > > > > > > > > > > > > > > > > > > > > > recently
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > very
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see
> > more
> > > >> > people
> > > >> > > > > > willing
> > > >> > > > > > > to
> > > >> > > > > > > > > > > > > contribute to
> > > >> > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > ecosystem.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before
> any
> > > >> > further
> > > >> > > > > > > > discussion,
> > > >> > > > > > > > > > > would
> > > >> > > > > > > > > > > > > you
> > > >> > > > > > > > > > > > > > > > please
> > > >> > > > > > > > > > > > > > > > > > > give
> > > >> > > > > > > > > > > > > > > > > > > > > some
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > explanation
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the
> > > >> relationship
> > > >> > > > > between
> > > >> > > > > > > this
> > > >> > > > > > > > > > > thread
> > > >> > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > current
> > > >> > > > > > > > > > > > > > > > > > > > > existing
> > > >> > > > > > > > > > > > > > > > > > > > > > > > JIRAs
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source
> [1]
> > > and
> > > >> > sink
> > > >> > > > [2]
> > > >> > > > > > > > > > connector?
> > > >> > > > > > > > > > > > > Will the
> > > >> > > > > > > > > > > > > > > > > > > > > contribution
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > contains
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > part
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those
> PRs
> > or
> > > >> > > totally
> > > >> > > > > > > > different
> > > >> > > > > > > > > > > > > > > > implementation?
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > https://issues.apache.org/jira/browse/FLINK-9641
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > https://issues.apache.org/jira/browse/FLINK-9168
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > ________________________________
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From:
> > Yijie
> > > >> Shen
> > > >> > <
> > > >> > > > > > > > > > > > > > > [hidden email]>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent:
> > > Tuesday,
> > > >> > > > > September
> > > >> > > > > > 3,
> > > >> > > > > > > > > 2019
> > > >> > > > > > > > > > > > 13:57
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To:
> > > >> > > > > [hidden email]
> > > >> > > > > > <
> > > >> > > > > > > > > > > > > > > > [hidden email]
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject:
> > > >> > [DISCUSS]
> > > >> > > > > > > Contribute
> > > >> > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > > > back
> > > >> > > > > > > > > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear
> Flink
> > > >> > > Community!
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would
> > > like to
> > > >> > > open
> > > >> > > > > the
> > > >> > > > > > > > > > discussion
> > > >> > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > contributing
> > > >> > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> connector
> > > [0]
> > > >> > back
> > > >> > > to
> > > >> > > > > > > Flink.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A
> brief
> > > >> > > > introduction
> > > >> > > > > > to
> > > >> > > > > > > > > Apache
> > > >> > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache
> > > >> Pulsar[1]
> > > >> > > is a
> > > >> > > > > > > > > > multi-tenant,
> > > >> > > > > > > > > > > > > > > > > > > high-performance
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > distributed
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub
> > > >> messaging
> > > >> > > > > system.
> > > >> > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > includes
> > > >> > > > > > > > > > > > > > > > > > multiple
> > > >> > > > > > > > > > > > > > > > > > > > > > features
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > such
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native
> > > support
> > > >> > for
> > > >> > > > > > multiple
> > > >> > > > > > > > > > > clusters
> > > >> > > > > > > > > > > > > in a
> > > >> > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > instance,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless
> > > >> > > > > geo-replication
> > > >> > > > > > of
> > > >> > > > > > > > > > > messages
> > > >> > > > > > > > > > > > > across
> > > >> > > > > > > > > > > > > > > > > > > clusters,
> > > >> > > > > > > > > > > > > > > > > > > > > > very
> > > >> > > > > > > > > > > > > > > > > > > > > > > > low
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and
> > > end-to-end
> > > >> > > > latency,
> > > >> > > > > > > > > seamless
> > > >> > > > > > > > > > > > > > > scalability
> > > >> > > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > over a
> > > >> > > > > > > > > > > > > > > > > > > > > > > > million
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and
> > > guaranteed
> > > >> > > > message
> > > >> > > > > > > > delivery
> > > >> > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > persistent
> > > >> > > > > > > > > > > > > > > > > > > message
> > > >> > > > > > > > > > > > > > > > > > > > > > > > storage
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided
> > by
> > > >> > Apache
> > > >> > > > > > > > BookKeeper.
> > > >> > > > > > > > > > > > > Nowadays,
> > > >> > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > has
> > > >> > > > > > > > > > > > > > > > > > > > > been
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > adopted
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > by
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and
> > > more
> > > >> > > > > > companies[2].
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The
> > > status
> > > >> of
> > > >> > > > Pulsar
> > > >> > > > > > > Flink
> > > >> > > > > > > > > > > > connector
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The
> Pulsar
> > > >> Flink
> > > >> > > > > > connector
> > > >> > > > > > > we
> > > >> > > > > > > > > are
> > > >> > > > > > > > > > > > > planning
> > > >> > > > > > > > > > > > > > > to
> > > >> > > > > > > > > > > > > > > > > > > > > contribute
> > > >> > > > > > > > > > > > > > > > > > > > > > is
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > built
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink
> > 1.9.0
> > > and
> > > >> > > > Pulsar
> > > >> > > > > > > 2.4.0.
> > > >> > > > > > > > > The
> > > >> > > > > > > > > > > > main
> > > >> > > > > > > > > > > > > > > > features
> > > >> > > > > > > > > > > > > > > > > > > are:
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar
> > as
> > > a
> > > >> > > > streaming
> > > >> > > > > > > > source
> > > >> > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > exactly-once
> > > >> > > > > > > > > > > > > > > > > > > > > > guarantee.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink
> > > >> streaming
> > > >> > > > > results
> > > >> > > > > > to
> > > >> > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > > > > > at-least-once
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > semantics.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > (We
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would
> > update
> > > >> this
> > > >> > > to
> > > >> > > > > > > > > exactly-once
> > > >> > > > > > > > > > > as
> > > >> > > > > > > > > > > > > well
> > > >> > > > > > > > > > > > > > > > when
> > > >> > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > gets
> > > >> > > > > > > > > > > > > > > > > > > > > > > > all
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > transaction
> > > >> > > features
> > > >> > > > > > ready
> > > >> > > > > > > in
> > > >> > > > > > > > > its
> > > >> > > > > > > > > > > > 2.5.0
> > > >> > > > > > > > > > > > > > > > > version)
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build
> > upon
> > > >> > Flink
> > > >> > > > new
> > > >> > > > > > > Table
> > > >> > > > > > > > > API
> > > >> > > > > > > > > > > Type
> > > >> > > > > > > > > > > > > > > system
> > > >> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]),
> > > >> > > > > > > > > > > > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > can
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > automatically
> > > >> > > > > > (de)serialize
> > > >> > > > > > > > > > > messages
> > > >> > > > > > > > > > > > > with
> > > >> > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > help
> > > >> > > > > > > > > > > > > > > > > > > of
> > > >> > > > > > > > > > > > > > > > > > > > > > > Pulsar
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> -
> > Integrate
> > > >> with
> > > >> > > > Flink
> > > >> > > > > > new
> > > >> > > > > > > > > > Catalog
> > > >> > > > > > > > > > > > API
> > > >> > > > > > > > > > > > > > > > > > > (FLIP-30[4]),
> > > >> > > > > > > > > > > > > > > > > > > > > > which
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > enables
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of
> > > Pulsar
> > > >> > > topics
> > > >> > > > as
> > > >> > > > > > > > tables
> > > >> > > > > > > > > in
> > > >> > > > > > > > > > > > > Table API
> > > >> > > > > > > > > > > > > > > > as
> > > >> > > > > > > > > > > > > > > > > > > well as
> > > >> > > > > > > > > > > > > > > > > > > > > > SQL
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > client.
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ##
> > Reference
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0]
> > > >> > > > > > > > > > > > > > > > >
> > > >> https://github.com/streamnative/pulsar-flink
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1]
> > > >> > > > > > > > https://pulsar.apache.org/
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2]
> > > >> > > > > > > > > > > > > > >
> https://pulsar.apache.org/en/powered-by/
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3]
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4]
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best,
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie
> Shen
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>>
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>

[Sijie Guo-2]

+1 for starting with pulsar sink and catalog.

Sijie

On Tue, Sep 24, 2019 at 3:36 PM Jark Wu <[hidden email]> wrote:

> I agree that we can start from Pulsar sink and Pulsar Catalog.
> When we finish that, maybe FLIP-27 is ready then.
>
> Best,
> Jark
>
> On Tue, 24 Sep 2019 at 17:06, Becket Qin <[hidden email]> wrote:
>
>> Hi Till,
>>
>> My understanding is that we are going to hold back a little bit on the
>> Pulsar source connector until FLIP-27 is ready. If we check in the old
>> Pulsar source connector right away, we will have to maintain that old
>> connector for quite a while (a year or more), which is something we don't
>> want to do.
>>
>> So the conclusion is following:
>> - We will try our best to check in both FLIP-27 and the Pulsar source
>> connector in 1.10.
>> - In case there isn't enough time to get Pulsar source connector checked
>> in
>> 1.10, we will link prominently to the external Pulsar connector repo in
>> our
>> connector doc so that users can find that easily.
>>
>> BTW, I think we can start the work on Pulsar sink connector given that
>> there is no clear plan on whether or how we are going to change the sink
>> interface.
>>
>> @Sijie, @Yijie, what do you think?
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Tue, Sep 24, 2019 at 4:40 PM Till Rohrmann <[hidden email]>
>> wrote:
>>
>> > What is the conclusion of this discussion? Are we gonna merge the old
>> > source/sink or do we believe that FLIP-27 will be completed in time so
>> that
>> > we can merge the new source right away?
>> >
>> > Cheers,
>> > Till
>> >
>> > On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <[hidden email]> wrote:
>> >
>> > >
>> > >
>> > > unscribe
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > At 2019-09-24 08:53:25, "Becket Qin" <[hidden email]> wrote:
>> > > >Thanks, Stephan.
>> > > >
>> > > >Sounds good to me. We can still try our best to get new Pulsar
>> connector
>> > > in
>> > > >Flink 1.10. In case we do not have time to do that, we will
>> prominently
>> > > >link the Pulsar connector from the Flink connector docs.
>> > > >
>> > > >Thanks,
>> > > >
>> > > >Jiangjie (Becket) Qin
>> > > >
>> > > >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <[hidden email]>
>> wrote:
>> > > >
>> > > >> Okay, I see your point, Becket.
>> > > >>
>> > > >> Then let us prominently link the Pulsar connector from the Flink
>> > > connector
>> > > >> docs then, so that users can find it easily.
>> > > >>
>> > > >> As soon as FLIP 27 is done, we reach out the Pulsar folks to
>> > contribute
>> > > a
>> > > >> new connector.
>> > > >>
>> > > >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <[hidden email]>
>> > > wrote:
>> > > >>
>> > > >> > Hi Stephan,
>> > > >> >
>> > > >> > I have no doubt about the value of adding Pulsar connector to
>> Flink
>> > > repo.
>> > > >> > My concern is about how exactly we are going to do it.
>> > > >> >
>> > > >> > As mentioned before, I believe that we can handle connectors more
>> > > >> > > pragmatically and less strict than the core of Flink, if it
>> helps
>> > > >> > unlocking
>> > > >> > > users.
>> > > >> >
>> > > >> > I can see the benefit of being less restrict for the initial
>> > connector
>> > > >> code
>> > > >> > adoption. However, I don't think we should be less restrict on
>> the
>> > > >> > maintenance commitment once the code is in Flink repo. It only
>> makes
>> > > >> sense
>> > > >> > to check in something and ask users to use if we plan to maintain
>> > it.
>> > > >> >
>> > > >> > If I understand correctly, the current plan so far is following:
>> > > >> > 1. release 1.10
>> > > >> >    - Check in Pulsar connector on old interface and label it as
>> beta
>> > > >> > version.
>> > > >> >    - encourage users to try it and report bugs.
>> > > >> > 2. release 1.11
>> > > >> >    - Check in Pulsar connector on new interface (a.k.a new Pulsar
>> > > >> > connector) and label it as beta version
>> > > >> >    - Deprecate the old Pulsar connector
>> > > >> >    - Fix bugs reported on old Pulsar connector from release 1.10
>> > > >> >    - Ask users to migrate from old Pulsar connector to new Pulsar
>> > > >> connector
>> > > >> > 3. release 1.12
>> > > >> >    - Announce end of support for old Pulsar connector and remove
>> the
>> > > code
>> > > >> >    - Fix bugs reported on new Pulsar connector.
>> > > >> >
>> > > >> > If this is the plan, it seems neither Flink nor the users trying
>> the
>> > > old
>> > > >> > Pulsar connector will benefit from this experimental old Pulsar
>> > > >> connector,
>> > > >> > because whatever feedbacks we got or bugs we fix on the old
>> Pulsar
>> > > >> > connector are immediately thrown away in one or two releases.
>> > > >> >
>> > > >> > If we check in the old Pulsar connector right now, the only
>> option I
>> > > see
>> > > >> is
>> > > >> > to maintain it for a while (e.g. a year or more). IMO, the
>> immediate
>> > > >> > deprecation and code removal hurts the users much more than
>> asking
>> > > them
>> > > >> to
>> > > >> > wait for another release. I personally think that we can avoid
>> this
>> > > >> > maintenance burden by going directly to the new Pulsar connector,
>> > > >> > especially given that users can still use the connector even if
>> they
>> > > are
>> > > >> > not in Flink repo. That said, I am OK with maintaining both old
>> and
>> > > new
>> > > >> > Pulsar connector if we believe that having the Pulsar connector
>> > > available
>> > > >> > right now in Flink repo is more important.
>> > > >> >
>> > > >> > Thanks,
>> > > >> >
>> > > >> > Jiangjie (Becket) Qin
>> > > >> >
>> > > >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <[hidden email]>
>> > > wrote:
>> > > >> >
>> > > >> > > My assumption is as Sijie's, that once the connector is either
>> > part
>> > > of
>> > > >> > > Flink, or part of the streamnative repo. No double maintenance.
>> > > >> > >
>> > > >> > > I feel this discussion is very much caught in problems that are
>> > all
>> > > >> > > solvable if we want to solve them.
>> > > >> > > Maybe we can think what our goal for users and the communities
>> is?
>> > > >> > >
>> > > >> > >   - Do we want to help build a relationship between the Pulsar
>> and
>> > > >> Flink
>> > > >> > > open source communities?
>> > > >> > >   - Will users find a connector in the streamnative repository?
>> > > >> > >   - Will users trust a connector that is not part of Flink as
>> > much?
>> > > >> > >
>> > > >> > > And then decide what is best according to the overall goals
>> there.
>> > > >> > > As mentioned before, I believe that we can handle connectors
>> more
>> > > >> > > pragmatically and less strict than the core of Flink, if it
>> helps
>> > > >> > unlocking
>> > > >> > > users.
>> > > >> > >
>> > > >> > > Best,
>> > > >> > > Stephan
>> > > >> > >
>> > > >> > >
>> > > >> > >
>> > > >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <[hidden email]>
>> > > wrote:
>> > > >> > >
>> > > >> > > > Thanks Becket.
>> > > >> > > >
>> > > >> > > > I think it is better for the Flink community to judge the
>> > > benefits of
>> > > >> > > doing
>> > > >> > > > this. I was trying to provide some views from outsiders.
>> > > >> > > >
>> > > >> > > > Thanks,
>> > > >> > > > Sijie
>> > > >> > > >
>> > > >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <
>> > [hidden email]
>> > > >
>> > > >> > > wrote:
>> > > >> > > >
>> > > >> > > > > Hi Sijie,
>> > > >> > > > >
>> > > >> > > > > Yes, we will have to support existing old connectors and
>> new
>> > > >> > connectors
>> > > >> > > > in
>> > > >> > > > > parallel for a while. We have to take that maintenance
>> > overhead
>> > > >> > because
>> > > >> > > > > existing connectors have been used by the users for a long
>> > > time. I
>> > > >> > > guess
>> > > >> > > > It
>> > > >> > > > > may take at least a year for us to fully remove the old
>> > > connectors.
>> > > >> > > > >
>> > > >> > > > > Process wise, we can do the same for Pulsar connector. But
>> I
>> > am
>> > > not
>> > > >> > > sure
>> > > >> > > > if
>> > > >> > > > > we want to have the same burden on Pulsar connector, and I
>> > would
>> > > >> like
>> > > >> > > to
>> > > >> > > > > understand the benefit of doing that.
>> > > >> > > > >
>> > > >> > > > > For users, the benefit of having the old Pulsar connector
>> > > checked
>> > > >> in
>> > > >> > > > seems
>> > > >> > > > > limited because 1) that code base will be immediately
>> > > deprecated in
>> > > >> > the
>> > > >> > > > > next release in 3-4 months; 2) users can always use it
>> even if
>> > > it
>> > > >> is
>> > > >> > > not
>> > > >> > > > in
>> > > >> > > > > the Flink code base. Admittedly it is not as convenient as
>> > > having
>> > > >> it
>> > > >> > in
>> > > >> > > > > Flink code base, but doesn't seem super either. And after
>> 3-4
>> > > >> months,
>> > > >> > > > users
>> > > >> > > > > can just use the new connector in Flink repo.
>> > > >> > > > >
>> > > >> > > > > For Flink developers, the old connector code base is not
>> > > something
>> > > >> > that
>> > > >> > > > we
>> > > >> > > > > want to evolve later. Instead, these code will be
>> deprecated
>> > and
>> > > >> > > > > removed. So why do we want to get a beta version out to
>> > attract
>> > > >> > people
>> > > >> > > to
>> > > >> > > > > use something we don't want to maintain?
>> > > >> > > > >
>> > > >> > > > > Thanks,
>> > > >> > > > >
>> > > >> > > > > Jiangjie (Becket) Qin
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <
>> > [hidden email]>
>> > > >> > wrote:
>> > > >> > > > >
>> > > >> > > > > > Thanks everyone here. Sorry for jumping into the
>> discussion
>> > > here.
>> > > >> > > > > >
>> > > >> > > > > > I am not very familiar about the deprecation process in
>> > Flink.
>> > > >> If I
>> > > >> > > > > > misunderstood the process, please fix me.
>> > > >> > > > > >
>> > > >> > > > > > As far as I understand, FLIP-27 is introducing a new
>> unified
>> > > API
>> > > >> > for
>> > > >> > > > > > connectors. After it introduces the new API
>> > > >> > > > > > and before moving all the existing connectors from old
>> API
>> > to
>> > > new
>> > > >> > > API,
>> > > >> > > > > both
>> > > >> > > > > > old ApI and new API will co-exist
>> > > >> > > > > > for a while until Flink moves all existing connectors to
>> new
>> > > API.
>> > > >> > So
>> > > >> > > > the
>> > > >> > > > > > Pulsar connector (using old API) can
>> > > >> > > > > > follow the deprecation process with other connector using
>> > old
>> > > API
>> > > >> > and
>> > > >> > > > the
>> > > >> > > > > > deprecation of old API, no?
>> > > >> > > > > >
>> > > >> > > > > > If that's the case, I think contributing the current
>> > connector
>> > > >> back
>> > > >> > > to
>> > > >> > > > > > Flink rather than maintaining it outside Flink
>> > > >> > > > > > would provide a bit more benefits. We can deprecate the
>> > > existing
>> > > >> > > > > > streamnative/pulsar-flink repo and point the users
>> > > >> > > > > > to use the connector in Flink repo. So all the review
>> > > processes
>> > > >> > will
>> > > >> > > > > happen
>> > > >> > > > > > within Flink for both old connector and
>> > > >> > > > > > new connector. It also reduces the confusions for the
>> users
>> > as
>> > > >> the
>> > > >> > > > > > documentation and code base happen in one place.
>> > > >> > > > > >
>> > > >> > > > > > Thoughts?
>> > > >> > > > > >
>> > > >> > > > > > - Sijie
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin <
>> > > >> [hidden email]>
>> > > >> > > > > wrote:
>> > > >> > > > > >
>> > > >> > > > > > > Thanks for the explanation, Stephan. I have a few
>> > questions
>> > > /
>> > > >> > > > thoughts.
>> > > >> > > > > > >
>> > > >> > > > > > > So that means we will remove the old connector without
>> a
>> > > major
>> > > >> > > > version
>> > > >> > > > > > > bump, is that correct?
>> > > >> > > > > > >
>> > > >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11
>> > > >> connectors
>> > > >> > > > will
>> > > >> > > > > > > always work because we saw some dependency class
>> > collisions
>> > > in
>> > > >> > the
>> > > >> > > > > past.
>> > > >> > > > > > To
>> > > >> > > > > > > make it safe we may have to maintain the old code for
>> one
>> > > more
>> > > >> > > > release.
>> > > >> > > > > > >
>> > > >> > > > > > > To be honest I am still wondering if we have to put the
>> > old
>> > > >> > > connector
>> > > >> > > > > in
>> > > >> > > > > > > Flink repo. if we check in the old connector to Flink.
>> We
>> > > will
>> > > >> > end
>> > > >> > > up
>> > > >> > > > > in
>> > > >> > > > > > > the following situation:
>> > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
>> > > >> > > > > > > 2. Old connector in Flink Repo, which may be different
>> > from
>> > > the
>> > > >> > one
>> > > >> > > > in
>> > > >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11,
>> removed
>> > in
>> > > >> 1.12)
>> > > >> > > > > > > 3. New connector in Flink Repo.
>> > > >> > > > > > >
>> > > >> > > > > > > We need to think about how to make the users in each
>> case
>> > > >> happy.
>> > > >> > > > > > > - For users of (1), I assume Sijie and Yijie will have
>> to
>> > > >> > maintain
>> > > >> > > > the
>> > > >> > > > > > code
>> > > >> > > > > > > a bit longer for its own compatibility even after we
>> have
>> > > (2).
>> > > >> In
>> > > >> > > > that
>> > > >> > > > > > > case, bugs found in old connector may or may not need
>> to
>> > be
>> > > >> fixed
>> > > >> > > in
>> > > >> > > > > both
>> > > >> > > > > > > Flink and the streamnative/pulsar-flink repo.
>> > > >> > > > > > > - For users of (2), will we provide bug fixes? If we
>> do,
>> > it
>> > > >> will
>> > > >> > > be a
>> > > >> > > > > > > little awkward because those bug fixes will be
>> immediately
>> > > >> > > deprecated
>> > > >> > > > > in
>> > > >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking
>> > > users
>> > > >> to
>> > > >> > > > > migrate
>> > > >> > > > > > > away from the bug fix. After Flink 1.12, users may
>> still
>> > > have
>> > > >> to
>> > > >> > > > switch
>> > > >> > > > > > to
>> > > >> > > > > > > use (3) due to the potential dependency class conflicts
>> > > >> mentioned
>> > > >> > > > > above.
>> > > >> > > > > > > - Users of (3) have a much easier life and don't need
>> to
>> > > worry
>> > > >> > too
>> > > >> > > > > much.
>> > > >> > > > > > >
>> > > >> > > > > > > The above story seems a little complicated to tell. I
>> > think
>> > > it
>> > > >> > will
>> > > >> > > > be
>> > > >> > > > > > much
>> > > >> > > > > > > easier to not have (2) at all.
>> > > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo.
>> > > >> > > > > > > 3. New connector in Flink Repo.
>> > > >> > > > > > >
>> > > >> > > > > > > - Old connector will only be maintained in
>> > > >> > > streamnative/pulsar-flink
>> > > >> > > > > repo
>> > > >> > > > > > > until it is fully deprecated. Users can always use the
>> > > existing
>> > > >> > > > Pulsar
>> > > >> > > > > > > connector in that repo.
>> > > >> > > > > > > - New connector will be in Flink repo and maintained
>> like
>> > > the
>> > > >> > other
>> > > >> > > > > > > connectors.
>> > > >> > > > > > >
>> > > >> > > > > > > This seems much simpler for users understand and they
>> will
>> > > not
>> > > >> be
>> > > >> > > > > blocked
>> > > >> > > > > > > from using the old connector. If the concern is about
>> the
>> > > >> quality
>> > > >> > > of
>> > > >> > > > > the
>> > > >> > > > > > > connector in streamnative/pulsar-flink repo, is it
>> enough
>> > > for
>> > > >> us
>> > > >> > > just
>> > > >> > > > > to
>> > > >> > > > > > > review the code in streamnative/pulsar-flink connector
>> to
>> > > make
>> > > >> > sure
>> > > >> > > > it
>> > > >> > > > > > > looks good from Flink's perspective?
>> > > >> > > > > > >
>> > > >> > > > > > > What do you think?
>> > > >> > > > > > >
>> > > >> > > > > > > Thanks,
>> > > >> > > > > > >
>> > > >> > > > > > > Jiangjie (Becket) Qin
>> > > >> > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen <
>> > > [hidden email]
>> > > >> >
>> > > >> > > > wrote:
>> > > >> > > > > > >
>> > > >> > > > > > > > My take would be the following:
>> > > >> > > > > > > >
>> > > >> > > > > > > >   - If we merge the connector now and replace it
>> with a
>> > > >> FLIP-27
>> > > >> > > > > version
>> > > >> > > > > > > > before the 1.10 release, then we need no deprecation
>> > > process
>> > > >> > > > > > > >   - If we don't manage to replace it with a FLIP-27
>> > > version
>> > > >> > > before
>> > > >> > > > > the
>> > > >> > > > > > > 1.10
>> > > >> > > > > > > > release, than it is good that we have the other
>> version,
>> > > so
>> > > >> no
>> > > >> > > > users
>> > > >> > > > > > get
>> > > >> > > > > > > > blocked.
>> > > >> > > > > > > >
>> > > >> > > > > > > > In the latter case we can see how we want to do it.
>> > > Immediate
>> > > >> > > > removal
>> > > >> > > > > > of
>> > > >> > > > > > > > the old version or deprecation label and keeping it
>> for
>> > > one
>> > > >> > more
>> > > >> > > > > > release.
>> > > >> > > > > > > > Given that you should be able to use a Flink 1.10
>> > > connector
>> > > >> > with
>> > > >> > > > > Flink
>> > > >> > > > > > > 1.11
>> > > >> > > > > > > > as well (stable public APIs) there is also a
>> workaround
>> > if
>> > > >> you
>> > > >> > > need
>> > > >> > > > > an
>> > > >> > > > > > > old
>> > > >> > > > > > > > connector in a newer version. So immediate removal
>> might
>> > > even
>> > > >> > be
>> > > >> > > > > > > feasible.
>> > > >> > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin <
>> > > >> > > [hidden email]>
>> > > >> > > > > > > wrote:
>> > > >> > > > > > > >
>> > > >> > > > > > > > > Hi Stephan,
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Thanks for the clarification. I completely agree
>> with
>> > > you
>> > > >> and
>> > > >> > > > > Thomas
>> > > >> > > > > > on
>> > > >> > > > > > > > the
>> > > >> > > > > > > > > process of adding connectors to Flink repo.
>> However, I
>> > > am
>> > > >> > > > wondering
>> > > >> > > > > > > what
>> > > >> > > > > > > > is
>> > > >> > > > > > > > > the deprecation process? Given the main concern
>> here
>> > was
>> > > >> that
>> > > >> > > we
>> > > >> > > > > may
>> > > >> > > > > > > have
>> > > >> > > > > > > > > to maintain two Pulsar connector code bases until
>> the
>> > > old
>> > > >> one
>> > > >> > > is
>> > > >> > > > > > > removed
>> > > >> > > > > > > > > from the repo, it would be good to know how long we
>> > > have to
>> > > >> > do
>> > > >> > > > > that.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Thanks,
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Jiangjie (Becket) Qin
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen <
>> > > >> > [hidden email]
>> > > >> > > >
>> > > >> > > > > > wrote:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > > Some quick thoughts on the connector contribution
>> > > >> process.
>> > > >> > I
>> > > >> > > > > > > basically
>> > > >> > > > > > > > > > reiterate here what Thomas mentioned in another
>> > thread
>> > > >> > about
>> > > >> > > > the
>> > > >> > > > > > > > Kinesis
>> > > >> > > > > > > > > > connector.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > For connectors, we should favor a low-overhead
>> > > >> contribution
>> > > >> > > > > > process,
>> > > >> > > > > > > > and
>> > > >> > > > > > > > > > accept user code and changes more readily than in
>> > the
>> > > >> core
>> > > >> > > > > system.
>> > > >> > > > > > > > > > That is because connectors have both a big
>> variety
>> > of
>> > > >> > > scenarios
>> > > >> > > > > > they
>> > > >> > > > > > > > get
>> > > >> > > > > > > > > > used in (only through use and many small
>> > > contributions do
>> > > >> > > they
>> > > >> > > > > > become
>> > > >> > > > > > > > > > really useful over time) and at the same time,
>> and
>> > > >> > committers
>> > > >> > > > do
>> > > >> > > > > > not
>> > > >> > > > > > > > use
>> > > >> > > > > > > > > > the connector themselves and usually cannot
>> foresee
>> > > too
>> > > >> > well
>> > > >> > > > what
>> > > >> > > > > > is
>> > > >> > > > > > > > > > needed.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Further more, a missing connector (or connector
>> > > feature)
>> > > >> is
>> > > >> > > > > often a
>> > > >> > > > > > > > > bigger
>> > > >> > > > > > > > > > show stopper for users than a missing API or
>> system
>> > > >> > feature.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Along these lines of thougt, the conclusion
>> would be
>> > > to
>> > > >> > take
>> > > >> > > > the
>> > > >> > > > > > > Pulsar
>> > > >> > > > > > > > > > connector now, focus the review on
>> > > >> legal/dependencies/rough
>> > > >> > > > code
>> > > >> > > > > > > style
>> > > >> > > > > > > > > and
>> > > >> > > > > > > > > > conventions, label it as "beta" (in the sense of
>> > "new
>> > > >> code"
>> > > >> > > > that
>> > > >> > > > > is
>> > > >> > > > > > > > "not
>> > > >> > > > > > > > > > yet tested through longer use") and go ahead. And
>> > then
>> > > >> > evolve
>> > > >> > > > it
>> > > >> > > > > > > > quickly
>> > > >> > > > > > > > > > without putting formal blockers in the way,
>> meaning
>> > > also
>> > > >> > > > adding a
>> > > >> > > > > > new
>> > > >> > > > > > > > > FLIP
>> > > >> > > > > > > > > > 27 version when it is there.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Best,
>> > > >> > > > > > > > > > Stephan
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin <
>> > > >> > > > [hidden email]
>> > > >> > > > > >
>> > > >> > > > > > > > wrote:
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > > Hi Yijie,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Could you please follow the FLIP process to
>> start
>> > a
>> > > new
>> > > >> > > FLIP
>> > > >> > > > > > > > > [DISCUSSION]
>> > > >> > > > > > > > > > > thread in the mailing list?
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing
>> list
>> > > now.
>> > > >> So
>> > > >> > > > there
>> > > >> > > > > > is a
>> > > >> > > > > > > > > FLIP
>> > > >> > > > > > > > > > > number collision. Can you change the FLIP
>> number
>> > to
>> > > 72?
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Jiangjie (Becket) Qin
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong <
>> > > >> > > > > [hidden email]>
>> > > >> > > > > > > > > wrote:
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > > Hi Yijie,
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP.
>> > > >> > > > > > > > > > > > Would you mind enabling comments/suggestions
>> on
>> > > the
>> > > >> > > google
>> > > >> > > > > doc
>> > > >> > > > > > > > link?
>> > > >> > > > > > > > > > This
>> > > >> > > > > > > > > > > > way the contributors from the community can
>> > > comment
>> > > >> on
>> > > >> > > the
>> > > >> > > > > doc.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > Best,
>> > > >> > > > > > > > > > > > Rong
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen <
>> > > >> > > > > > > > > [hidden email]>
>> > > >> > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Hello everyone,
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > I've drafted a FLIP that describes the
>> current
>> > > >> design
>> > > >> > > of
>> > > >> > > > > the
>> > > >> > > > > > > > Pulsar
>> > > >> > > > > > > > > > > > > connector:
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Please take a look and let me know what you
>> > > think.
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > Yijie
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong
>> <
>> > > >> > > > > > > [hidden email]>
>> > > >> > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Hi All,
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Sorry for joining the discussion late and
>> > > thanks
>> > > >> > > Yijie
>> > > >> > > > &
>> > > >> > > > > > > Sijie
>> > > >> > > > > > > > > for
>> > > >> > > > > > > > > > > > > driving
>> > > >> > > > > > > > > > > > > > the discussion.
>> > > >> > > > > > > > > > > > > > I also think the Pulsar connector would
>> be a
>> > > very
>> > > >> > > > > valuable
>> > > >> > > > > > > > > addition
>> > > >> > > > > > > > > > > to
>> > > >> > > > > > > > > > > > > > Flink. I can also help out a bit on the
>> > review
>> > > >> side
>> > > >> > > :-)
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Regarding the timeline, I also share
>> > concerns
>> > > >> with
>> > > >> > > > Becket
>> > > >> > > > > > on
>> > > >> > > > > > > > the
>> > > >> > > > > > > > > > > > > > relationship between the new Pulsar
>> > connector
>> > > and
>> > > >> > > > > FLIP-27.
>> > > >> > > > > > > > > > > > > > There's also another discussion just
>> started
>> > > by
>> > > >> > > Stephan
>> > > >> > > > > on
>> > > >> > > > > > > > > dropping
>> > > >> > > > > > > > > > > > Kafka
>> > > >> > > > > > > > > > > > > > 9/10 support on next Flink release [1].
>> > > Although
>> > > >> > the
>> > > >> > > > > > > situation
>> > > >> > > > > > > > > is
>> > > >> > > > > > > > > > > > > somewhat
>> > > >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has
>> been
>> > > in
>> > > >> > Flink
>> > > >> > > > for
>> > > >> > > > > > > > almost
>> > > >> > > > > > > > > > 3-4
>> > > >> > > > > > > > > > > > > years,
>> > > >> > > > > > > > > > > > > > based on the discussion I am not sure if
>> a
>> > > major
>> > > >> > > > version
>> > > >> > > > > > > > release
>> > > >> > > > > > > > > > is a
>> > > >> > > > > > > > > > > > > > requirement for removing old connector
>> > > supports.
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > I think there shouldn't be a blocker if
>> we
>> > > agree
>> > > >> > the
>> > > >> > > > old
>> > > >> > > > > > > > > connector
>> > > >> > > > > > > > > > > will
>> > > >> > > > > > > > > > > > > be
>> > > >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar
>> connector
>> > is
>> > > >> > there.
>> > > >> > > > As
>> > > >> > > > > > > > Stephan
>> > > >> > > > > > > > > > > > stated,
>> > > >> > > > > > > > > > > > > it
>> > > >> > > > > > > > > > > > > > is easier to contribute the source sooner
>> > and
>> > > >> > adjust
>> > > >> > > it
>> > > >> > > > > > > later.
>> > > >> > > > > > > > > > > > > > We should also ensure we clearly
>> communicate
>> > > the
>> > > >> > > > message:
>> > > >> > > > > > for
>> > > >> > > > > > > > > > > example,
>> > > >> > > > > > > > > > > > > > putting an experimental flag on the
>> > pre-FLIP27
>> > > >> > > > connector
>> > > >> > > > > > page
>> > > >> > > > > > > > of
>> > > >> > > > > > > > > > the
>> > > >> > > > > > > > > > > > > > website, documentations, etc. Any other
>> > > thoughts?
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > --
>> > > >> > > > > > > > > > > > > > Rong
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > [1]
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > >
>> >
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket
>> Qin <
>> > > >> > > > > > > > [hidden email]
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > Technically speaking, removing the old
>> > > >> connector
>> > > >> > > code
>> > > >> > > > > is
>> > > >> > > > > > a
>> > > >> > > > > > > > > > > backwards
>> > > >> > > > > > > > > > > > > > > incompatible change which requires a
>> major
>> > > >> > version
>> > > >> > > > > bump,
>> > > >> > > > > > > i.e.
>> > > >> > > > > > > > > > Flink
>> > > >> > > > > > > > > > > > > 2.x.
>> > > >> > > > > > > > > > > > > > > Given that we don't have a clear plan
>> on
>> > > when
>> > > >> to
>> > > >> > > have
>> > > >> > > > > the
>> > > >> > > > > > > > next
>> > > >> > > > > > > > > > > major
>> > > >> > > > > > > > > > > > > > > version release, it seems unclear how
>> long
>> > > the
>> > > >> > old
>> > > >> > > > > > > connector
>> > > >> > > > > > > > > code
>> > > >> > > > > > > > > > > > will
>> > > >> > > > > > > > > > > > > be
>> > > >> > > > > > > > > > > > > > > there if we check it in right now. Or
>> will
>> > > we
>> > > >> > > remove
>> > > >> > > > > the
>> > > >> > > > > > > old
>> > > >> > > > > > > > > > > > connector
>> > > >> > > > > > > > > > > > > > > without a major version bump? In any
>> case,
>> > > it
>> > > >> > > sounds
>> > > >> > > > > not
>> > > >> > > > > > > > quite
>> > > >> > > > > > > > > > user
>> > > >> > > > > > > > > > > > > > > friendly to the those who might use the
>> > old
>> > > >> > Pulsar
>> > > >> > > > > > > > connector. I
>> > > >> > > > > > > > > > am
>> > > >> > > > > > > > > > > > not
>> > > >> > > > > > > > > > > > > sure
>> > > >> > > > > > > > > > > > > > > if it is worth these potential
>> problems in
>> > > >> order
>> > > >> > to
>> > > >> > > > > have
>> > > >> > > > > > > the
>> > > >> > > > > > > > > > Pulsar
>> > > >> > > > > > > > > > > > > source
>> > > >> > > > > > > > > > > > > > > connector checked in one or two months
>> > > earlier.
>> > > >> > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > >> > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan
>> > > Ewen <
>> > > >> > > > > > > > [hidden email]
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > Agreed, if we check in the old code,
>> we
>> > > >> should
>> > > >> > > make
>> > > >> > > > > it
>> > > >> > > > > > > > clear
>> > > >> > > > > > > > > > that
>> > > >> > > > > > > > > > > > it
>> > > >> > > > > > > > > > > > > will
>> > > >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27
>> based
>> > > >> version
>> > > >> > > of
>> > > >> > > > > the
>> > > >> > > > > > > > > > connector
>> > > >> > > > > > > > > > > is
>> > > >> > > > > > > > > > > > > > > there.
>> > > >> > > > > > > > > > > > > > > > We should not commit to maintaining
>> the
>> > > old
>> > > >> > > > version,
>> > > >> > > > > > that
>> > > >> > > > > > > > > would
>> > > >> > > > > > > > > > > be
>> > > >> > > > > > > > > > > > > indeed
>> > > >> > > > > > > > > > > > > > > > too much overhead.
>> > > >> > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM
>> Becket
>> > > Qin <
>> > > >> > > > > > > > > > [hidden email]
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > Hi Stephan,
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > Thanks for the volunteering to
>> help.
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > Yes, the overhead would just be
>> review
>> > > >> > > capacity.
>> > > >> > > > In
>> > > >> > > > > > > > fact, I
>> > > >> > > > > > > > > > am
>> > > >> > > > > > > > > > > > not
>> > > >> > > > > > > > > > > > > > > > worrying
>> > > >> > > > > > > > > > > > > > > > > too much about the review capacity.
>> > > That is
>> > > >> > > just
>> > > >> > > > a
>> > > >> > > > > > one
>> > > >> > > > > > > > time
>> > > >> > > > > > > > > > > cost.
>> > > >> > > > > > > > > > > > > My
>> > > >> > > > > > > > > > > > > > > > > concern is mainly about the long
>> term
>> > > >> burden.
>> > > >> > > > > Assume
>> > > >> > > > > > we
>> > > >> > > > > > > > > have
>> > > >> > > > > > > > > > > new
>> > > >> > > > > > > > > > > > > source
>> > > >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly
>> > added
>> > > >> > Pulsar
>> > > >> > > > > > > > connectors
>> > > >> > > > > > > > > in
>> > > >> > > > > > > > > > > old
>> > > >> > > > > > > > > > > > > > > > > interface. Later on if we migrate
>> > > Pulsar to
>> > > >> > new
>> > > >> > > > > > source
>> > > >> > > > > > > > > > > interface,
>> > > >> > > > > > > > > > > > > the
>> > > >> > > > > > > > > > > > > > > old
>> > > >> > > > > > > > > > > > > > > > > Pulsar connector might be
>> deprecated
>> > > almost
>> > > >> > > > > > immediately
>> > > >> > > > > > > > > after
>> > > >> > > > > > > > > > > > > checked
>> > > >> > > > > > > > > > > > > > > in,
>> > > >> > > > > > > > > > > > > > > > > but we may still have to maintain
>> two
>> > > code
>> > > >> > > bases.
>> > > >> > > > > For
>> > > >> > > > > > > the
>> > > >> > > > > > > > > > > > existing
>> > > >> > > > > > > > > > > > > > > > > connectors, we have to do that
>> > anyways.
>> > > But
>> > > >> > it
>> > > >> > > > > would
>> > > >> > > > > > be
>> > > >> > > > > > > > > good
>> > > >> > > > > > > > > > to
>> > > >> > > > > > > > > > > > > avoid
>> > > >> > > > > > > > > > > > > > > > > introducing a new connector with
>> the
>> > > same
>> > > >> > > > problem.
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM
>> > Stephan
>> > > >> Ewen
>> > > >> > <
>> > > >> > > > > > > > > > [hidden email]
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > Hi all!
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > Nice to see this lively
>> discussion
>> > > about
>> > > >> > the
>> > > >> > > > > Pulsar
>> > > >> > > > > > > > > > > connector.
>> > > >> > > > > > > > > > > > > > > > > > Some thoughts on the open
>> questions:
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or
>> maintain
>> > as
>> > > a
>> > > >> > > > community
>> > > >> > > > > > > > package
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > Looks like the discussion is more
>> > > going
>> > > >> > > towards
>> > > >> > > > > > > > > > > contribution. I
>> > > >> > > > > > > > > > > > > think
>> > > >> > > > > > > > > > > > > > > > > that
>> > > >> > > > > > > > > > > > > > > > > > is good, especially if we think
>> that
>> > > we
>> > > >> > want
>> > > >> > > to
>> > > >> > > > > > > build a
>> > > >> > > > > > > > > > > > similarly
>> > > >> > > > > > > > > > > > > > > deep
>> > > >> > > > > > > > > > > > > > > > > > integration with Pulsar as we
>> have
>> > for
>> > > >> > > example
>> > > >> > > > > with
>> > > >> > > > > > > > > Kafka.
>> > > >> > > > > > > > > > > The
>> > > >> > > > > > > > > > > > > > > > connector
>> > > >> > > > > > > > > > > > > > > > > > already looks like a more
>> thorough
>> > > >> > connector
>> > > >> > > > than
>> > > >> > > > > > > many
>> > > >> > > > > > > > > > others
>> > > >> > > > > > > > > > > > we
>> > > >> > > > > > > > > > > > > have
>> > > >> > > > > > > > > > > > > > > > in
>> > > >> > > > > > > > > > > > > > > > > > the repository.
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > With either a repo split, or the
>> new
>> > > >> build
>> > > >> > > > > system,
>> > > >> > > > > > I
>> > > >> > > > > > > > hope
>> > > >> > > > > > > > > > > that
>> > > >> > > > > > > > > > > > > the
>> > > >> > > > > > > > > > > > > > > > build
>> > > >> > > > > > > > > > > > > > > > > > overhead is not a problem.
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > ## Committer Support
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > Becket offered some help
>> already, I
>> > > can
>> > > >> > also
>> > > >> > > > > help a
>> > > >> > > > > > > > bit.
>> > > >> > > > > > > > > I
>> > > >> > > > > > > > > > > hope
>> > > >> > > > > > > > > > > > > that
>> > > >> > > > > > > > > > > > > > > > > > between us, we can cover this.
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for
>> > FLIP-27
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is
>> actually
>> > > >> making
>> > > >> > > some
>> > > >> > > > > > > PoC-ing
>> > > >> > > > > > > > > > > > > progress, but
>> > > >> > > > > > > > > > > > > > > > > will
>> > > >> > > > > > > > > > > > > > > > > > take 2 more months, I would
>> > estimate,
>> > > >> > before
>> > > >> > > it
>> > > >> > > > > is
>> > > >> > > > > > > > fully
>> > > >> > > > > > > > > > > > > available.
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > If we want to be on the safe side
>> > with
>> > > >> the
>> > > >> > > > > > > > contribution,
>> > > >> > > > > > > > > we
>> > > >> > > > > > > > > > > > > should
>> > > >> > > > > > > > > > > > > > > > > > contribute the source sooner and
>> > > adjust
>> > > >> it
>> > > >> > > > later.
>> > > >> > > > > > > That
>> > > >> > > > > > > > > > would
>> > > >> > > > > > > > > > > > also
>> > > >> > > > > > > > > > > > > > > help
>> > > >> > > > > > > > > > > > > > > > us
>> > > >> > > > > > > > > > > > > > > > > > in case things get crazy towards
>> the
>> > > 1.10
>> > > >> > > > feature
>> > > >> > > > > > > > freeze
>> > > >> > > > > > > > > > and
>> > > >> > > > > > > > > > > it
>> > > >> > > > > > > > > > > > > would
>> > > >> > > > > > > > > > > > > > > > be
>> > > >> > > > > > > > > > > > > > > > > > hard to find time to review the
>> new
>> > > >> > changes.
>> > > >> > > > > > > > > > > > > > > > > > What would be the overhead of
>> > > >> contributing
>> > > >> > > now?
>> > > >> > > > > > Given
>> > > >> > > > > > > > > that
>> > > >> > > > > > > > > > > the
>> > > >> > > > > > > > > > > > > code
>> > > >> > > > > > > > > > > > > > > is
>> > > >> > > > > > > > > > > > > > > > > > already there, it looks like it
>> > would
>> > > be
>> > > >> > only
>> > > >> > > > > > review
>> > > >> > > > > > > > > > > capacity,
>> > > >> > > > > > > > > > > > > right?
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > Best,
>> > > >> > > > > > > > > > > > > > > > > > Stephan
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM
>> > Yijie
>> > > >> > Shen <
>> > > >> > > > > > > > > > > > > > > [hidden email]
>> > > >> > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > Hi everyone!
>> > > >> > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > Thanks for your attention and
>> the
>> > > >> > promotion
>> > > >> > > > of
>> > > >> > > > > > this
>> > > >> > > > > > > > > work.
>> > > >> > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon
>> as
>> > > >> > possible
>> > > >> > > > for
>> > > >> > > > > > more
>> > > >> > > > > > > > > > > specific
>> > > >> > > > > > > > > > > > > > > > > discussions.
>> > > >> > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we
>> have
>> > > not
>> > > >> > > > reached
>> > > >> > > > > a
>> > > >> > > > > > > > > > consensus.
>> > > >> > > > > > > > > > > > > > > > Therefore,
>> > > >> > > > > > > > > > > > > > > > > > > I will explain all the
>> > > functionalities
>> > > >> of
>> > > >> > > the
>> > > >> > > > > > > > existing
>> > > >> > > > > > > > > > > > > connector in
>> > > >> > > > > > > > > > > > > > > > > > > the FLIP (including Source,
>> Sink,
>> > > and
>> > > >> > > > Catalog)
>> > > >> > > > > to
>> > > >> > > > > > > > > > continue
>> > > >> > > > > > > > > > > > our
>> > > >> > > > > > > > > > > > > > > > > > > discussions in FLIP.
>> > > >> > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > Thanks for your kind help.
>> > > >> > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > Best,
>> > > >> > > > > > > > > > > > > > > > > > > Yijie
>> > > >> > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM
>> > > Becket
>> > > >> > Qin
>> > > >> > > <
>> > > >> > > > > > > > > > > > > [hidden email]>
>> > > >> > > > > > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > Hi Sijie,
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is
>> to
>> > > have
>> > > >> > > Pulsar
>> > > >> > > > > > > > connector
>> > > >> > > > > > > > > > in
>> > > >> > > > > > > > > > > > > 1.10,
>> > > >> > > > > > > > > > > > > > > how
>> > > >> > > > > > > > > > > > > > > > > > about
>> > > >> > > > > > > > > > > > > > > > > > > we
>> > > >> > > > > > > > > > > > > > > > > > > > do the following:
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar
>> > > >> connector
>> > > >> > > to
>> > > >> > > > > > Flink
>> > > >> > > > > > > > main
>> > > >> > > > > > > > > > > repo
>> > > >> > > > > > > > > > > > > as it
>> > > >> > > > > > > > > > > > > > > > is a
>> > > >> > > > > > > > > > > > > > > > > > new
>> > > >> > > > > > > > > > > > > > > > > > > > public interface to Flink
>> main
>> > > repo.
>> > > >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar
>> > sink
>> > > >> > right
>> > > >> > > > away
>> > > >> > > > > > as
>> > > >> > > > > > > > > there
>> > > >> > > > > > > > > > is
>> > > >> > > > > > > > > > > > no
>> > > >> > > > > > > > > > > > > > > change
>> > > >> > > > > > > > > > > > > > > > > to
>> > > >> > > > > > > > > > > > > > > > > > > the
>> > > >> > > > > > > > > > > > > > > > > > > > sink interface so far.
>> > > >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on
>> FLIP-27.
>> > > >> Flink
>> > > >> > > 1.10
>> > > >> > > > > is
>> > > >> > > > > > > > going
>> > > >> > > > > > > > > to
>> > > >> > > > > > > > > > > be
>> > > >> > > > > > > > > > > > > code
>> > > >> > > > > > > > > > > > > > > > > freeze
>> > > >> > > > > > > > > > > > > > > > > > in
>> > > >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we
>> give a
>> > > >> month
>> > > >> > to
>> > > >> > > > the
>> > > >> > > > > > > > > > development
>> > > >> > > > > > > > > > > > and
>> > > >> > > > > > > > > > > > > > > > review
>> > > >> > > > > > > > > > > > > > > > > of
>> > > >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to
>> > have
>> > > >> > FLIP-27
>> > > >> > > > by
>> > > >> > > > > > late
>> > > >> > > > > > > > > Oct.
>> > > >> > > > > > > > > > > > There
>> > > >> > > > > > > > > > > > > are
>> > > >> > > > > > > > > > > > > > > > > still
>> > > >> > > > > > > > > > > > > > > > > > 7
>> > > >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it
>> is
>> > > >> doable.
>> > > >> > > If
>> > > >> > > > > > > FLIP-27
>> > > >> > > > > > > > is
>> > > >> > > > > > > > > > not
>> > > >> > > > > > > > > > > > > ready
>> > > >> > > > > > > > > > > > > > > by
>> > > >> > > > > > > > > > > > > > > > > late
>> > > >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check
>> in
>> > > >> Pulsar
>> > > >> > > > > > connector
>> > > >> > > > > > > > with
>> > > >> > > > > > > > > > the
>> > > >> > > > > > > > > > > > > > > existing
>> > > >> > > > > > > > > > > > > > > > > > source
>> > > >> > > > > > > > > > > > > > > > > > > > interface. This means we will
>> > have
>> > > >> > Pulsar
>> > > >> > > > > > > connector
>> > > >> > > > > > > > > in
>> > > >> > > > > > > > > > > > Flink
>> > > >> > > > > > > > > > > > > > > 1.10,
>> > > >> > > > > > > > > > > > > > > > > > either
>> > > >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27.
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > Because we are going to have
>> > > Pulsar
>> > > >> > sink
>> > > >> > > > and
>> > > >> > > > > > > source
>> > > >> > > > > > > > > > > checked
>> > > >> > > > > > > > > > > > > in
>> > > >> > > > > > > > > > > > > > > > > > > separately,
>> > > >> > > > > > > > > > > > > > > > > > > > it might make sense to have
>> two
>> > > >> FLIPs,
>> > > >> > > one
>> > > >> > > > > for
>> > > >> > > > > > > > Pulsar
>> > > >> > > > > > > > > > > sink
>> > > >> > > > > > > > > > > > > and
>> > > >> > > > > > > > > > > > > > > > > another
>> > > >> > > > > > > > > > > > > > > > > > > for
>> > > >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can
>> start
>> > > the
>> > > >> > work
>> > > >> > > on
>> > > >> > > > > > > Pulsar
>> > > >> > > > > > > > > sink
>> > > >> > > > > > > > > > > > right
>> > > >> > > > > > > > > > > > > > > away.
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13
>> PM
>> > > Sijie
>> > > >> > Guo
>> > > >> > > <
>> > > >> > > > > > > > > > > > [hidden email]
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket.
>> > > >> > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink
>> > > >> community?
>> > > >> > > > Shall
>> > > >> > > > > > we
>> > > >> > > > > > > > wait
>> > > >> > > > > > > > > > for
>> > > >> > > > > > > > > > > > > FLIP-27
>> > > >> > > > > > > > > > > > > > > > or
>> > > >> > > > > > > > > > > > > > > > > > > shall we
>> > > >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And
>> > what
>> > > the
>> > > >> > > next
>> > > >> > > > > > steps
>> > > >> > > > > > > > are?
>> > > >> > > > > > > > > > :-)
>> > > >> > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > > > > > > > > > Sijie
>> > > >> > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at
>> 2:43 PM
>> > > >> Bowen
>> > > >> > > Li <
>> > > >> > > > > > > > > > > > > [hidden email]>
>> > > >> > > > > > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > > Hi,
>> > > >> > > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar
>> > > connector
>> > > >> > in
>> > > >> > > > > Flink
>> > > >> > > > > > > can
>> > > >> > > > > > > > > be a
>> > > >> > > > > > > > > > > > good
>> > > >> > > > > > > > > > > > > > > mutual
>> > > >> > > > > > > > > > > > > > > > > > > benefit
>> > > >> > > > > > > > > > > > > > > > > > > > > to
>> > > >> > > > > > > > > > > > > > > > > > > > > > both communities.
>> > > >> > > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > > Another perspective is
>> that
>> > > >> Pulsar
>> > > >> > > > > > connector
>> > > >> > > > > > > is
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > > > 1st
>> > > >> > > > > > > > > > > > > > > > streaming
>> > > >> > > > > > > > > > > > > > > > > > > > > connector
>> > > >> > > > > > > > > > > > > > > > > > > > > > that integrates with
>> Flink's
>> > > >> > metadata
>> > > >> > > > > > > > management
>> > > >> > > > > > > > > > > system
>> > > >> > > > > > > > > > > > > and
>> > > >> > > > > > > > > > > > > > > > > Catalog
>> > > >> > > > > > > > > > > > > > > > > > > APIs.
>> > > >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how
>> the
>> > > >> > > > integration
>> > > >> > > > > > > turns
>> > > >> > > > > > > > > out
>> > > >> > > > > > > > > > > and
>> > > >> > > > > > > > > > > > > > > whether
>> > > >> > > > > > > > > > > > > > > > we
>> > > >> > > > > > > > > > > > > > > > > > > need to
>> > > >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog
>> stack,
>> > > >> which
>> > > >> > > are
>> > > >> > > > > > > > currently
>> > > >> > > > > > > > > in
>> > > >> > > > > > > > > > > > > Beta, to
>> > > >> > > > > > > > > > > > > > > > > cater
>> > > >> > > > > > > > > > > > > > > > > > to
>> > > >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink.
>> Thus
>> > > I'm
>> > > >> in
>> > > >> > > > favor
>> > > >> > > > > of
>> > > >> > > > > > > > > merging
>> > > >> > > > > > > > > > > > > Pulsar
>> > > >> > > > > > > > > > > > > > > > > > connector
>> > > >> > > > > > > > > > > > > > > > > > > into
>> > > >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10.
>> > > >> > > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit
>> > smaller
>> > > >> sized
>> > > >> > > > PRs,
>> > > >> > > > > > e.g.
>> > > >> > > > > > > > > maybe
>> > > >> > > > > > > > > > > one
>> > > >> > > > > > > > > > > > > for
>> > > >> > > > > > > > > > > > > > > > basic
>> > > >> > > > > > > > > > > > > > > > > > > > > > source/sink
>> functionalities
>> > > and
>> > > >> > > another
>> > > >> > > > > for
>> > > >> > > > > > > > > schema
>> > > >> > > > > > > > > > > and
>> > > >> > > > > > > > > > > > > > > catalog
>> > > >> > > > > > > > > > > > > > > > > > > > > integration,
>> > > >> > > > > > > > > > > > > > > > > > > > > > just to make them easier
>> to
>> > > >> review.
>> > > >> > > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt
>> to
>> > > wait
>> > > >> for
>> > > >> > > > > > FLIP-27.
>> > > >> > > > > > > > But
>> > > >> > > > > > > > > I
>> > > >> > > > > > > > > > > > don't
>> > > >> > > > > > > > > > > > > > > think
>> > > >> > > > > > > > > > > > > > > > > > > FLIP-27
>> > > >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in
>> cases
>> > > >> where
>> > > >> > it
>> > > >> > > > > > cannot
>> > > >> > > > > > > > make
>> > > >> > > > > > > > > > its
>> > > >> > > > > > > > > > > > way
>> > > >> > > > > > > > > > > > > > > into
>> > > >> > > > > > > > > > > > > > > > > 1.10
>> > > >> > > > > > > > > > > > > > > > > > > or
>> > > >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable
>> > > amount
>> > > >> of
>> > > >> > > time
>> > > >> > > > > for
>> > > >> > > > > > > > > > > committers
>> > > >> > > > > > > > > > > > to
>> > > >> > > > > > > > > > > > > > > > review
>> > > >> > > > > > > > > > > > > > > > > or
>> > > >> > > > > > > > > > > > > > > > > > > for
>> > > >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully
>> > > adapt
>> > > >> to
>> > > >> > > new
>> > > >> > > > > > > > > interfaces.
>> > > >> > > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > > > > > Bowen
>> > > >> > > > > > > > > > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > > > > > >
>
>