FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Stephan Ewen
Hi all!

We just published a blog post about how streaming fault tolerance
mechanisms evolved, and what kind of performance Flink gets with its
checkpointing mechanism.

I think it is a pretty interesting read for people that are interested in
Flink or data streaming in general.

The blog post talks about:

  - Fault tolerance techniques, starting from acknowledgements, over micro
batches, to transactional updates and distributed snapshots.

  - Performance of Flink, throughput, latency, and tradeoffs.

  - A "chaos monkey" experiment where computation continues strongly
consistent even when periodically killing workers.


Comments welcome!

Greetings,
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Stephan Ewen
I forgot the link ;-)

http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/

On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:

> Hi all!
>
> We just published a blog post about how streaming fault tolerance
> mechanisms evolved, and what kind of performance Flink gets with its
> checkpointing mechanism.
>
> I think it is a pretty interesting read for people that are interested in
> Flink or data streaming in general.
>
> The blog post talks about:
>
>   - Fault tolerance techniques, starting from acknowledgements, over micro
> batches, to transactional updates and distributed snapshots.
>
>   - Performance of Flink, throughput, latency, and tradeoffs.
>
>   - A "chaos monkey" experiment where computation continues strongly
> consistent even when periodically killing workers.
>
>
> Comments welcome!
>
> Greetings,
> Stephan
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

hawin
Great job, Guys

Let me read it carefully.







On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <[hidden email]> wrote:

> I forgot the link ;-)
>
>
> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:
>
>> Hi all!
>>
>> We just published a blog post about how streaming fault tolerance
>> mechanisms evolved, and what kind of performance Flink gets with its
>> checkpointing mechanism.
>>
>> I think it is a pretty interesting read for people that are interested in
>> Flink or data streaming in general.
>>
>> The blog post talks about:
>>
>>   - Fault tolerance techniques, starting from acknowledgements, over
>> micro batches, to transactional updates and distributed snapshots.
>>
>>   - Performance of Flink, throughput, latency, and tradeoffs.
>>
>>   - A "chaos monkey" experiment where computation continues strongly
>> consistent even when periodically killing workers.
>>
>>
>> Comments welcome!
>>
>> Greetings,
>> Stephan
>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

ankurcha
Pretty awesome piece.

Sent from my iPhone

> On Aug 5, 2015, at 10:10, Hawin Jiang <[hidden email]> wrote:
>
> Great job, Guys
>
> Let me read it carefully.
>
>
>
>
>
>
>
>> On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen <[hidden email]> wrote:
>> I forgot the link ;-)
>>
>> http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
>>
>>> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:
>>> Hi all!
>>>
>>> We just published a blog post about how streaming fault tolerance mechanisms evolved, and what kind of performance Flink gets with its checkpointing mechanism.
>>>
>>> I think it is a pretty interesting read for people that are interested in Flink or data streaming in general.
>>>
>>> The blog post talks about:
>>>
>>>   - Fault tolerance techniques, starting from acknowledgements, over micro batches, to transactional updates and distributed snapshots.
>>>
>>>   - Performance of Flink, throughput, latency, and tradeoffs.
>>>
>>>   - A "chaos monkey" experiment where computation continues strongly consistent even when periodically killing workers.
>>>
>>>
>>> Comments welcome!
>>>
>>> Greetings,
>>> Stephan
>
Reply | Threaded
Open this post in threaded view
|

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Theodore Vasiloudis
In reply to this post by Stephan Ewen
Great post Stephan! A small note: the code for Google Dataflow does display
correctly for me, I'm getting &lt and &gt instead of <>

On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:

> Hi all!
>
> We just published a blog post about how streaming fault tolerance
> mechanisms evolved, and what kind of performance Flink gets with its
> checkpointing mechanism.
>
> I think it is a pretty interesting read for people that are interested in
> Flink or data streaming in general.
>
> The blog post talks about:
>
>   - Fault tolerance techniques, starting from acknowledgements, over micro
> batches, to transactional updates and distributed snapshots.
>
>   - Performance of Flink, throughput, latency, and tradeoffs.
>
>   - A "chaos monkey" experiment where computation continues strongly
> consistent even when periodically killing workers.
>
>
> Comments welcome!
>
> Greetings,
> Stephan
>
Reply | Threaded
Open this post in threaded view
|

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Sachin Goel
This makes me want to dig into the streaming API. About time perhaps. :')

Cheers!
Sachin

-- Sachin Goel
Computer Science, IIT Delhi
m. +91-9871457685
On Aug 6, 2015 2:39 AM, "Theodore Vasiloudis" <
[hidden email]> wrote:

> Great post Stephan! A small note: the code for Google Dataflow does display
> correctly for me, I'm getting &lt and &gt instead of <>
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:
>
> > Hi all!
> >
> > We just published a blog post about how streaming fault tolerance
> > mechanisms evolved, and what kind of performance Flink gets with its
> > checkpointing mechanism.
> >
> > I think it is a pretty interesting read for people that are interested in
> > Flink or data streaming in general.
> >
> > The blog post talks about:
> >
> >   - Fault tolerance techniques, starting from acknowledgements, over
> micro
> > batches, to transactional updates and distributed snapshots.
> >
> >   - Performance of Flink, throughput, latency, and tradeoffs.
> >
> >   - A "chaos monkey" experiment where computation continues strongly
> > consistent even when periodically killing workers.
> >
> >
> > Comments welcome!
> >
> > Greetings,
> > Stephan
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Stephan Ewen
In reply to this post by Theodore Vasiloudis
Thanks, Theo.

Fixed the markup...

On Wed, Aug 5, 2015 at 11:08 PM, Theodore Vasiloudis <
[hidden email]> wrote:

> Great post Stephan! A small note: the code for Google Dataflow does display
> correctly for me, I'm getting &lt and &gt instead of <>
>
> On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen <[hidden email]> wrote:
>
> > Hi all!
> >
> > We just published a blog post about how streaming fault tolerance
> > mechanisms evolved, and what kind of performance Flink gets with its
> > checkpointing mechanism.
> >
> > I think it is a pretty interesting read for people that are interested in
> > Flink or data streaming in general.
> >
> > The blog post talks about:
> >
> >   - Fault tolerance techniques, starting from acknowledgements, over
> micro
> > batches, to transactional updates and distributed snapshots.
> >
> >   - Performance of Flink, throughput, latency, and tradeoffs.
> >
> >   - A "chaos monkey" experiment where computation continues strongly
> > consistent even when periodically killing workers.
> >
> >
> > Comments welcome!
> >
> > Greetings,
> > Stephan
> >
>