Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io
We're trying to get metrics such as throughput to filesystem sinks. Are the metrics listed on that page automatically recorded, and we just need to retrieve them? If so, would BucketingSink be the place to add metrics to be visible in Prometheus? |
Hi,
the metrics listed on the web page are registered for all tasks automatically. Thus, you should be able to simply consume them by configuring the respective Prometheus metric reporter. One thing to note is that the metrics are reported per Task and not per logical operator. Thus, if your source is chained with another upstream operator, then you can retrieve the metrics for the chain of these operators. Cheers, Till On Tue, Feb 13, 2018 at 8:37 PM, cw7k <[hidden email]> wrote: > Hi, couple questions on IO metrics listed here:https://ci.apache.org/ > projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io > > We're trying to get metrics such as throughput to filesystem sinks. Are > the metrics listed on that page automatically recorded, and we just need to > retrieve them? If so, would BucketingSink be the place to add metrics to > be visible in Prometheus? |
In reply to this post by cw7k
All metrics listed are automatically measured, you only need to
configure the Prometheus reporter. Note that we do not measure how much data a source is reading / a sink is writing (see FLINK-7286 <https://issues.apache.org/jira/browse/FLINK-7286>). If you want to measure these you will have to modify the source code of the respective source/sink classes, like the BucketingSink. Do make sure that your custom metrics have different names than the built-in ones as otherwise the will be ignored. On 13.02.2018 20:37, cw7k wrote: > Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io > > We're trying to get metrics such as throughput to filesystem sinks. Are the metrics listed on that page automatically recorded, and we just need to retrieve them? If so, would BucketingSink be the place to add metrics to be visible in Prometheus? |
To use the "numBytesOutPerSecond" example, what exactly is being measured? Is there an example app with usage of this metric?
Just to clarify, if I register this meter in BucketingSink, it will be ignored? Does this mean I need to implement my own measurement mechanism in BucketingSink and set up a new meter for throughput? Sample use case is a sink to the local filesystem and to cloud storage. On Wednesday, February 14, 2018, 1:10:47 AM PST, Chesnay Schepler <[hidden email]> wrote: All metrics listed are automatically measured, you only need to configure the Prometheus reporter. Note that we do not measure how much data a source is reading / a sink is writing (see FLINK-7286 <https://issues.apache.org/jira/browse/FLINK-7286>). If you want to measure these you will have to modify the source code of the respective source/sink classes, like the BucketingSink. Do make sure that your custom metrics have different names than the built-in ones as otherwise the will be ignored. On 13.02.2018 20:37, cw7k wrote: > Hi, couple questions on IO metrics listed here:https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io > > We're trying to get metrics such as throughput to filesystem sinks. Are the metrics listed on that page automatically recorded, and we just need to retrieve them? If so, would BucketingSink be the place to add metrics to be visible in Prometheus? |
The metric `numButesOutPerSecond` measures the number of bytes the
corresponding task emits per second over the network to a consuming task. If the task is a sink, then this metric will be 0, because there is no down stream operator. If you want to have additional information from your BucketingSink, e.g. how many bytes are written to the file system, then you can register your own metrics by subclassing the sink. Cheers, Till On Wed, Feb 14, 2018 at 7:33 PM, cw7k <[hidden email]> wrote: > To use the "numBytesOutPerSecond" example, what exactly is being > measured? Is there an example app with usage of this metric? > > Just to clarify, if I register this meter in BucketingSink, it will be > ignored? Does this mean I need to implement my own measurement mechanism > in BucketingSink and set up a new meter for throughput? Sample use case is > a sink to the local filesystem and to cloud storage. > On Wednesday, February 14, 2018, 1:10:47 AM PST, Chesnay Schepler < > [hidden email]> wrote: > > All metrics listed are automatically measured, you only need to > configure the Prometheus reporter. > > Note that we do not measure how much data a source is reading / a sink > is writing (see FLINK-7286 > <https://issues.apache.org/jira/browse/FLINK-7286>). > If you want to measure these you will have to modify the source code of > the respective source/sink classes, > like the BucketingSink. Do make sure that your custom metrics have > different names than the built-in ones > as otherwise the will be ignored. > > On 13.02.2018 20:37, cw7k wrote: > > Hi, couple questions on IO metrics listed here:https://ci.apache.org/ > projects/flink/flink-docs-release-1.4/monitoring/metrics.html#io > > > > We're trying to get metrics such as throughput to filesystem sinks. Are > the metrics listed on that page automatically recorded, and we just need to > retrieve them? If so, would BucketingSink be the place to add metrics to > be visible in Prometheus? > > > > |
Free forum by Nabble | Edit this page |