IO metrics

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

IO metrics

cw7k
Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io

We're trying to get metrics such as throughput to filesystem sinks.  Are the metrics listed on that page automatically recorded, and we just need to retrieve them?  If so, would BucketingSink be the place to add metrics to be visible in Prometheus? 
Reply | Threaded
Open this post in threaded view
|

Re: IO metrics

Till Rohrmann
Hi,

the metrics listed on the web page are registered for all tasks
automatically. Thus, you should be able to simply consume them by
configuring the respective Prometheus metric reporter. One thing to note is
that the metrics are reported per Task and not per logical operator. Thus,
if your source is chained with another upstream operator, then you can
retrieve the metrics for the chain of these operators.

Cheers,
Till

On Tue, Feb 13, 2018 at 8:37 PM, cw7k <[hidden email]> wrote:

> Hi, couple questions on IO metrics listed here:https://ci.apache.org/
> projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io
>
> We're trying to get metrics such as throughput to filesystem sinks.  Are
> the metrics listed on that page automatically recorded, and we just need to
> retrieve them?  If so, would BucketingSink be the place to add metrics to
> be visible in Prometheus?
Reply | Threaded
Open this post in threaded view
|

Re: IO metrics

Chesnay Schepler-3
In reply to this post by cw7k
All metrics listed are automatically measured, you only need to
configure the Prometheus reporter.

Note that we do not measure how much data a source is reading / a sink
is writing (see FLINK-7286
<https://issues.apache.org/jira/browse/FLINK-7286>).
If you want to measure these you will have to modify the source code of
the respective source/sink classes,
like the BucketingSink. Do make sure that your custom metrics have
different names than the built-in ones
as otherwise the will be ignored.

On 13.02.2018 20:37, cw7k wrote:
> Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io
>
> We're trying to get metrics such as throughput to filesystem sinks.  Are the metrics listed on that page automatically recorded, and we just need to retrieve them?  If so, would BucketingSink be the place to add metrics to be visible in Prometheus?


Reply | Threaded
Open this post in threaded view
|

Re: IO metrics

cw7k
 To use the "numBytesOutPerSecond" example, what exactly is being measured?  Is there an example app with usage of this metric?

Just to clarify, if I register this meter in BucketingSink, it will be ignored?  Does this mean I need to implement my own measurement mechanism in BucketingSink and set up a new meter for throughput?  Sample use case is a sink to the local filesystem and to cloud storage.
    On Wednesday, February 14, 2018, 1:10:47 AM PST, Chesnay Schepler <[hidden email]> wrote:  
 
 All metrics listed are automatically measured, you only need to
configure the Prometheus reporter.

Note that we do not measure how much data a source is reading / a sink
is writing (see FLINK-7286
<https://issues.apache.org/jira/browse/FLINK-7286>).
If you want to measure these you will have to modify the source code of
the respective source/sink classes,
like the BucketingSink. Do make sure that your custom metrics have
different names than the built-in ones
as otherwise the will be ignored.

On 13.02.2018 20:37, cw7k wrote:
> Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io
>
> We're trying to get metrics such as throughput to filesystem sinks.  Are the metrics listed on that page automatically recorded, and we just need to retrieve them?  If so, would BucketingSink be the place to add metrics to be visible in Prometheus?


 
Reply | Threaded
Open this post in threaded view
|

Re: IO metrics

Till Rohrmann
The metric `numButesOutPerSecond` measures the number of  bytes the
corresponding task emits per second over the network to a consuming task.
If the task is a sink, then this metric will be 0, because there is no down
stream operator. If you want to have additional information from your
BucketingSink, e.g. how many bytes are written to the file system, then you
can register your own metrics by subclassing the sink.

Cheers,
Till

On Wed, Feb 14, 2018 at 7:33 PM, cw7k <[hidden email]> wrote:

>  To use the "numBytesOutPerSecond" example, what exactly is being
> measured?  Is there an example app with usage of this metric?
>
> Just to clarify, if I register this meter in BucketingSink, it will be
> ignored?  Does this mean I need to implement my own measurement mechanism
> in BucketingSink and set up a new meter for throughput?  Sample use case is
> a sink to the local filesystem and to cloud storage.
>     On Wednesday, February 14, 2018, 1:10:47 AM PST, Chesnay Schepler <
> [hidden email]> wrote:
>
>  All metrics listed are automatically measured, you only need to
> configure the Prometheus reporter.
>
> Note that we do not measure how much data a source is reading / a sink
> is writing (see FLINK-7286
> <https://issues.apache.org/jira/browse/FLINK-7286>).
> If you want to measure these you will have to modify the source code of
> the respective source/sink classes,
> like the BucketingSink. Do make sure that your custom metrics have
> different names than the built-in ones
> as otherwise the will be ignored.
>
> On 13.02.2018 20:37, cw7k wrote:
> > Hi, couple questions on IO metrics listed here:https://ci.apache.org/
> projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io
> >
> > We're trying to get metrics such as throughput to filesystem sinks.  Are
> the metrics listed on that page automatically recorded, and we just need to
> retrieve them?  If so, would BucketingSink be the place to add metrics to
> be visible in Prometheus?
>
>
>
>