Kafka producer exactly once

[Pramod Immaneni-2]

Is there a Kafka producer that can do exactly once semantic without the use
of transactions?

Thanks

[Till Rohrmann]

Hi Pramod,

Flink's Kafka connector uses transactions in order to support exactly once
semantic.

Cheers,
Till

On Thu, Jan 7, 2021 at 11:17 PM Pramod Immaneni <[hidden email]> wrote:

> Is there a Kafka producer that can do exactly once semantic without the use
> of transactions?
>
> Thanks
>

[Piotr Nowojski-4]

Hi Pramod,

Moreover I don't think there is a way to implement exactly once producer
without some use of transactions one way or another.

Best,
Piotrek

pt., 8 sty 2021 o 09:34 Till Rohrmann <[hidden email]> napisał(a):

> Hi Pramod,
>
> Flink's Kafka connector uses transactions in order to support exactly once
> semantic.
>
> Cheers,
> Till
>
> On Thu, Jan 7, 2021 at 11:17 PM Pramod Immaneni <[hidden email]> wrote:
>
> > Is there a Kafka producer that can do exactly once semantic without the
> use
> > of transactions?
> >
> > Thanks
> >
>

[Aljoscha Krettek-2]

On 2021/01/07 14:17, Pramod Immaneni wrote:
>Is there a Kafka producer that can do exactly once semantic without the use
>of transactions?

I'm afraid not right now. There are some ideas about using a WAL (write
ahead log) and then periodically "shipping" that to Kafka but nothing
concrete.

Best,
Aljoscha

[Aljoscha Krettek-2]

On 2021/01/08 10:00, Piotr Nowojski wrote:
>Moreover I don't think there is a way to implement exactly once producer
>without some use of transactions one way or another.

There are some ways I can think of. If messages have consistent IDs, we
could check whether a message is already in Kafka before writing it.  
That way, you could achieve exactly-once semantics with an
"at-least-once" system.

I'm not saying that it would be good, or high-performance, but I think
there are ways. 😅

Best,
Aljoscha

[Piotr Nowojski-5]

Yes, I meant true general purpose exactly-once :)

> There are some ideas about using a WAL (write ahead log) and then
periodically "shipping" that to Kafka but nothing concrete.

But that would still need to be using Kafka transactions for "shipping"
records. That's what I meant that one way or another we need transactions :)

Piotrek

pt., 8 sty 2021 o 12:44 Aljoscha Krettek <[hidden email]> napisał(a):

> On 2021/01/08 10:00, Piotr Nowojski wrote:
> >Moreover I don't think there is a way to implement exactly once producer
> >without some use of transactions one way or another.
>
> There are some ways I can think of. If messages have consistent IDs, we
> could check whether a message is already in Kafka before writing it.
> That way, you could achieve exactly-once semantics with an
> "at-least-once" system.
>
> I'm not saying that it would be good, or high-performance, but I think
> there are ways. 😅
>
> Best,
> Aljoscha
>