Hi all!
We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism. I think it is a pretty interesting read for people that are interested in Flink or data streaming in general. The blog post talks about: - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots. - Performance of Flink, throughput, latency, and tradeoffs. - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers. Comments welcome! Greetings, Stephan |
I forgot the link ;-)
http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote: > Hi all! > > We just published a blog post about how streaming fault tolerance > mechanisms evolved, and what kind of performance Flink gets with its > checkpointing mechanism. > > I think it is a pretty interesting read for people that are interested in > Flink or data streaming in general. > > The blog post talks about: > > - Fault tolerance techniques, starting from acknowledgements, over micro > batches, to transactional updates and distributed snapshots. > > - Performance of Flink, throughput, latency, and tradeoffs. > > - A "chaos monkey" experiment where computation continues strongly > consistent even when periodically killing workers. > > > Comments welcome! > > Greetings, > Stephan > > > |
Great job, Guys
Let me read it carefully. On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <[hidden email]> wrote: > I forgot the link ;-) > > > http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ > > On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote: > >> Hi all! >> >> We just published a blog post about how streaming fault tolerance >> mechanisms evolved, and what kind of performance Flink gets with its >> checkpointing mechanism. >> >> I think it is a pretty interesting read for people that are interested in >> Flink or data streaming in general. >> >> The blog post talks about: >> >> - Fault tolerance techniques, starting from acknowledgements, over >> micro batches, to transactional updates and distributed snapshots. >> >> - Performance of Flink, throughput, latency, and tradeoffs. >> >> - A "chaos monkey" experiment where computation continues strongly >> consistent even when periodically killing workers. >> >> >> Comments welcome! >> >> Greetings, >> Stephan >> >> >> > |
Pretty awesome piece.
Sent from my iPhone > On Aug 5, 2015, at 10:10, Hawin Jiang <[hidden email]> wrote: > > Great job, Guys > > Let me read it carefully. > > > > > > > >> On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <[hidden email]> wrote: >> I forgot the link ;-) >> >> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ >> >>> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote: >>> Hi all! >>> >>> We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism. >>> >>> I think it is a pretty interesting read for people that are interested in Flink or data streaming in general. >>> >>> The blog post talks about: >>> >>> - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots. >>> >>> - Performance of Flink, throughput, latency, and tradeoffs. >>> >>> - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers. >>> >>> >>> Comments welcome! >>> >>> Greetings, >>> Stephan > |
In reply to this post by Stephan Ewen
Great post Stephan! A small note: the code for Google Dataflow does display
correctly for me, I'm getting < and > instead of <> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote: > Hi all! > > We just published a blog post about how streaming fault tolerance > mechanisms evolved, and what kind of performance Flink gets with its > checkpointing mechanism. > > I think it is a pretty interesting read for people that are interested in > Flink or data streaming in general. > > The blog post talks about: > > - Fault tolerance techniques, starting from acknowledgements, over micro > batches, to transactional updates and distributed snapshots. > > - Performance of Flink, throughput, latency, and tradeoffs. > > - A "chaos monkey" experiment where computation continues strongly > consistent even when periodically killing workers. > > > Comments welcome! > > Greetings, > Stephan > |
This makes me want to dig into the streaming API. About time perhaps. :')
Cheers! Sachin -- Sachin Goel Computer Science, IIT Delhi m. +91-9871457685 On Aug 6, 2015 2:39 AM, "Theodore Vasiloudis" < [hidden email]> wrote: > Great post Stephan! A small note: the code for Google Dataflow does display > correctly for me, I'm getting < and > instead of <> > > On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote: > > > Hi all! > > > > We just published a blog post about how streaming fault tolerance > > mechanisms evolved, and what kind of performance Flink gets with its > > checkpointing mechanism. > > > > I think it is a pretty interesting read for people that are interested in > > Flink or data streaming in general. > > > > The blog post talks about: > > > > - Fault tolerance techniques, starting from acknowledgements, over > micro > > batches, to transactional updates and distributed snapshots. > > > > - Performance of Flink, throughput, latency, and tradeoffs. > > > > - A "chaos monkey" experiment where computation continues strongly > > consistent even when periodically killing workers. > > > > > > Comments welcome! > > > > Greetings, > > Stephan > > > |
In reply to this post by Theodore Vasiloudis
Thanks, Theo.
Fixed the markup... On Wed, Aug 5, 2015 at 11:08 PM, Theodore Vasiloudis < [hidden email]> wrote: > Great post Stephan! A small note: the code for Google Dataflow does display > correctly for me, I'm getting < and > instead of <> > > On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote: > > > Hi all! > > > > We just published a blog post about how streaming fault tolerance > > mechanisms evolved, and what kind of performance Flink gets with its > > checkpointing mechanism. > > > > I think it is a pretty interesting read for people that are interested in > > Flink or data streaming in general. > > > > The blog post talks about: > > > > - Fault tolerance techniques, starting from acknowledgements, over > micro > > batches, to transactional updates and distributed snapshots. > > > > - Performance of Flink, throughput, latency, and tradeoffs. > > > > - A "chaos monkey" experiment where computation continues strongly > > consistent even when periodically killing workers. > > > > > > Comments welcome! > > > > Greetings, > > Stephan > > > |
Free forum by Nabble | Edit this page |