Flink Accumulators vs Metrics

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flink Accumulators vs Metrics

Chawla,Sumit
Hi All

Based on my code reading,  I have following understanding of the Metrics
and Accumulators.

1.  Accumulators for a Flink JOB work like global counters.  They are
designed so that accumulator values from different instances of Execution
Vertex can be combined.  They are essentially distributed counters.

2.  Flink Metrics are local to Task Manager which is reporting those, and
need external aggregation for a Job centric view

I see that one can defined User Metrics as part of writing Flink Programs.
But these metrics would not be consolidated when the job is running same
task on different task managers.  Having said that,  Is it fair to classify
that Metrics are for surfacing operation details only, and would not be
replacing Accumulators anytime.

For my use case, I wanted to maintain some Global counters/ histograms. (
like the one available in Storm - e.g. Total Messages Processed in last 1
minute, last 10 minutes etc).   Metrics would have been perfect fit for
these but one would need to employ external aggregations to come up with
holistic view of metrics at JOB level.

Please correct my understanding if i am missing something here.


Regards
Sumit Chawla
Reply | Threaded
Open this post in threaded view
|

Re: Flink Accumulators vs Metrics

Chesnay Schepler-3
Hello,

you are correct that metrics are not aggregated on the Task- or
JobManager. Their primary purpose is to expose fine-grained information
to an external metric system, which could then do aggregations if desired.

There are plans to aggregate metrics in the WebInterface though, like
the total input per job.

Regards,
Chesnay

On 23.09.2016 00:48, Chawla,Sumit wrote:

> Hi All
>
> Based on my code reading,  I have following understanding of the Metrics
> and Accumulators.
>
> 1.  Accumulators for a Flink JOB work like global counters.  They are
> designed so that accumulator values from different instances of Execution
> Vertex can be combined.  They are essentially distributed counters.
>
> 2.  Flink Metrics are local to Task Manager which is reporting those, and
> need external aggregation for a Job centric view
>
> I see that one can defined User Metrics as part of writing Flink Programs.
> But these metrics would not be consolidated when the job is running same
> task on different task managers.  Having said that,  Is it fair to classify
> that Metrics are for surfacing operation details only, and would not be
> replacing Accumulators anytime.
>
> For my use case, I wanted to maintain some Global counters/ histograms. (
> like the one available in Storm - e.g. Total Messages Processed in last 1
> minute, last 10 minutes etc).   Metrics would have been perfect fit for
> these but one would need to employ external aggregations to come up with
> holistic view of metrics at JOB level.
>
> Please correct my understanding if i am missing something here.
>
>
> Regards
> Sumit Chawla
>