[DISCUSS] Standard / Convention for common connector metrics

[Becket Qin]

Hi folks,

I was trying to add some metrics to Kafka connectors and realized that
right now Flink does not have a common metric definition for the
connectors. This complicates the monitoring and operation because the
monitoring / alerts need to be set case by case.

To address this issue, I would like to see if is possible to have a set of
standardized common metrics for all sources and sinks. The following doc
describes the proposal. Feedback is very welcome.

https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#

Thanks,

Jiangjie (Becket) Qin

[Robert Metzger]

Hey Becket,
thanks a lot for your proposal!

Do you have an overview over the current situation of the metrics in the
connectors?
Which connectors expose metrics at all?
Are they different?

On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]> wrote:

> Hi folks,
>
> I was trying to add some metrics to Kafka connectors and realized that
> right now Flink does not have a common metric definition for the
> connectors. This complicates the monitoring and operation because the
> monitoring / alerts need to be set case by case.
>
> To address this issue, I would like to see if is possible to have a set of
> standardized common metrics for all sources and sinks. The following doc
> describes the proposal. Feedback is very welcome.
>
>
> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
>
> Thanks,
>
> Jiangjie (Becket) Qin
>

[Chesnay Schepler-3]

@Robert:
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors

On 31.01.2019 11:03, Robert Metzger wrote:

> Hey Becket,
> thanks a lot for your proposal!
>
> Do you have an overview over the current situation of the metrics in the
> connectors?
> Which connectors expose metrics at all?
> Are they different?
>
> On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]> wrote:
>
>> Hi folks,
>>
>> I was trying to add some metrics to Kafka connectors and realized that
>> right now Flink does not have a common metric definition for the
>> connectors. This complicates the monitoring and operation because the
>> monitoring / alerts need to be set case by case.
>>
>> To address this issue, I would like to see if is possible to have a set of
>> standardized common metrics for all sources and sinks. The following doc
>> describes the proposal. Feedback is very welcome.
>>
>>
>> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>

[Becket Qin]

Thanks for the connector metric url,  Chesnay :)

@Robert, as you can see, the metrics from different connectors are quite
different. And there are different names for similar metrics, which is a
little frustrating when users want to do monitoring / alerting.

Thanks,

Jiangjie (Becket) Qin

On Thu, Jan 31, 2019 at 6:41 PM Chesnay Schepler <[hidden email]> wrote:

> @Robert:
>
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors
>
> On 31.01.2019 11:03, Robert Metzger wrote:
> > Hey Becket,
> > thanks a lot for your proposal!
> >
> > Do you have an overview over the current situation of the metrics in the
> > connectors?
> > Which connectors expose metrics at all?
> > Are they different?
> >
> > On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]> wrote:
> >
> >> Hi folks,
> >>
> >> I was trying to add some metrics to Kafka connectors and realized that
> >> right now Flink does not have a common metric definition for the
> >> connectors. This complicates the monitoring and operation because the
> >> monitoring / alerts need to be set case by case.
> >>
> >> To address this issue, I would like to see if is possible to have a set
> of
> >> standardized common metrics for all sources and sinks. The following doc
> >> describes the proposal. Feedback is very welcome.
> >>
> >>
> >>
> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
>
>

[Konstantin Knauf-2]

Hi Becket,

I like the idea of providing a standard set of metrics, which sources can
choose to implement/expose. In addition, I think, sources, like the Kafka
or Kinesis Source, should continue to forward the original consumer metrics
under their original names, so that users familiar with Kafka/Kinesis can
relate to them easily.

Cheers,

Konstantin



On Fri, Feb 1, 2019 at 3:54 AM Becket Qin <[hidden email]> wrote:

> Thanks for the connector metric url,  Chesnay :)
>
> @Robert, as you can see, the metrics from different connectors are quite
> different. And there are different names for similar metrics, which is a
> little frustrating when users want to do monitoring / alerting.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Jan 31, 2019 at 6:41 PM Chesnay Schepler <[hidden email]>
> wrote:
>
> > @Robert:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors
> >
> > On 31.01.2019 11:03, Robert Metzger wrote:
> > > Hey Becket,
> > > thanks a lot for your proposal!
> > >
> > > Do you have an overview over the current situation of the metrics in
> the
> > > connectors?
> > > Which connectors expose metrics at all?
> > > Are they different?
> > >
> > > On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]>
> wrote:
> > >
> > >> Hi folks,
> > >>
> > >> I was trying to add some metrics to Kafka connectors and realized that
> > >> right now Flink does not have a common metric definition for the
> > >> connectors. This complicates the monitoring and operation because the
> > >> monitoring / alerts need to be set case by case.
> > >>
> > >> To address this issue, I would like to see if is possible to have a
> set
> > of
> > >> standardized common metrics for all sources and sinks. The following
> doc
> > >> describes the proposal. Feedback is very welcome.
> > >>
> > >>
> > >>
> >
> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
> > >>
> > >> Thanks,
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> >
> >
>

--
--

Konstantin Knauf | Solution Architect


--

Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany
<https://maps.google.com/?q=Stresemannstr.+121A,10963+Berlin,+Germany&entry=gmail&source=g>
data Artisans, Inc. | 1161 Mission Street, San Francisco, CA-94103, USA
<https://maps.google.com/?q=1161+Mission+Street,+San+Francisco,+CA-94103,+USA&entry=gmail&source=g>

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

[Becket Qin]

Thanks for the feedback, Konstantin,

Completely agree that exposing the original metrics from the clients of
other systems is a good practice and should be encouraged.

Thanks,

Jiangjie (Becket) Qin

On Fri, Feb 1, 2019 at 4:01 PM Konstantin Knauf <[hidden email]>
wrote:

> Hi Becket,
>
> I like the idea of providing a standard set of metrics, which sources can
> choose to implement/expose. In addition, I think, sources, like the Kafka
> or Kinesis Source, should continue to forward the original consumer metrics
> under their original names, so that users familiar with Kafka/Kinesis can
> relate to them easily.
>
> Cheers,
>
> Konstantin
>
>
>
> On Fri, Feb 1, 2019 at 3:54 AM Becket Qin <[hidden email]> wrote:
>
> > Thanks for the connector metric url,  Chesnay :)
> >
> > @Robert, as you can see, the metrics from different connectors are quite
> > different. And there are different names for similar metrics, which is a
> > little frustrating when users want to do monitoring / alerting.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Thu, Jan 31, 2019 at 6:41 PM Chesnay Schepler <[hidden email]>
> > wrote:
> >
> > > @Robert:
> > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors
> > >
> > > On 31.01.2019 11:03, Robert Metzger wrote:
> > > > Hey Becket,
> > > > thanks a lot for your proposal!
> > > >
> > > > Do you have an overview over the current situation of the metrics in
> > the
> > > > connectors?
> > > > Which connectors expose metrics at all?
> > > > Are they different?
> > > >
> > > > On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]>
> > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> I was trying to add some metrics to Kafka connectors and realized
> that
> > > >> right now Flink does not have a common metric definition for the
> > > >> connectors. This complicates the monitoring and operation because
> the
> > > >> monitoring / alerts need to be set case by case.
> > > >>
> > > >> To address this issue, I would like to see if is possible to have a
> > set
> > > of
> > > >> standardized common metrics for all sources and sinks. The following
> > doc
> > > >> describes the proposal. Feedback is very welcome.
> > > >>
> > > >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jiangjie (Becket) Qin
> > > >>
> > >
> > >
> >
>
>
> --
> --
>
> Konstantin Knauf | Solution Architect
>
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Data Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany
> <
> https://maps.google.com/?q=Stresemannstr.+121A,10963+Berlin,+Germany&entry=gmail&source=g
> >
> data Artisans, Inc. | 1161 Mission Street, San Francisco, CA-94103, USA
> <
> https://maps.google.com/?q=1161+Mission+Street,+San+Francisco,+CA-94103,+USA&entry=gmail&source=g
> >
>
> --
> Data Artisans GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>

[Shaoxuan Wang]

Becket,

Thanks for the proposal. I am much agree that we should only design the
standard/common Flink-related connector metrics, while keep the original
names for the ones provided by the connector itself. If everyone agrees to
make these changes, we should do it asap, the earlier to make the change
the less impact on the user's system. Maybe we can start to unify the
metrics on the existing connectors, and then slowly introduce the new
common/standard metrics if necessary.

Since metric is also a kind of user interface, we should either open a
Flip, or create a webpage to document the specifications, it's up to you.
The point is that we want to create a good document, such that the future
developers can easily find and follow these specifications.

Regards,
Shaoxuan

On Fri, Feb 1, 2019 at 5:15 PM Becket Qin <[hidden email]> wrote:

> Thanks for the feedback, Konstantin,
>
> Completely agree that exposing the original metrics from the clients of
> other systems is a good practice and should be encouraged.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Feb 1, 2019 at 4:01 PM Konstantin Knauf <
> [hidden email]>
> wrote:
>
> > Hi Becket,
> >
> > I like the idea of providing a standard set of metrics, which sources can
> > choose to implement/expose. In addition, I think, sources, like the Kafka
> > or Kinesis Source, should continue to forward the original consumer
> metrics
> > under their original names, so that users familiar with Kafka/Kinesis can
> > relate to them easily.
> >
> > Cheers,
> >
> > Konstantin
> >
> >
> >
> > On Fri, Feb 1, 2019 at 3:54 AM Becket Qin <[hidden email]> wrote:
> >
> > > Thanks for the connector metric url,  Chesnay :)
> > >
> > > @Robert, as you can see, the metrics from different connectors are
> quite
> > > different. And there are different names for similar metrics, which is
> a
> > > little frustrating when users want to do monitoring / alerting.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Thu, Jan 31, 2019 at 6:41 PM Chesnay Schepler <[hidden email]>
> > > wrote:
> > >
> > > > @Robert:
> > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors
> > > >
> > > > On 31.01.2019 11:03, Robert Metzger wrote:
> > > > > Hey Becket,
> > > > > thanks a lot for your proposal!
> > > > >
> > > > > Do you have an overview over the current situation of the metrics
> in
> > > the
> > > > > connectors?
> > > > > Which connectors expose metrics at all?
> > > > > Are they different?
> > > > >
> > > > > On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]>
> > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> I was trying to add some metrics to Kafka connectors and realized
> > that
> > > > >> right now Flink does not have a common metric definition for the
> > > > >> connectors. This complicates the monitoring and operation because
> > the
> > > > >> monitoring / alerts need to be set case by case.
> > > > >>
> > > > >> To address this issue, I would like to see if is possible to have
> a
> > > set
> > > > of
> > > > >> standardized common metrics for all sources and sinks. The
> following
> > > doc
> > > > >> describes the proposal. Feedback is very welcome.
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jiangjie (Becket) Qin
> > > > >>
> > > >
> > > >
> > >
> >
> >
> > --
> > --
> >
> > Konstantin Knauf | Solution Architect
> >
> >
> > --
> >
> > Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> > Conference
> >
> > Stream Processing | Event Driven | Real Time
> >
> > --
> >
> > Data Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany
> > <
> >
> https://maps.google.com/?q=Stresemannstr.+121A,10963+Berlin,+Germany&entry=gmail&source=g
> > >
> > data Artisans, Inc. | 1161 Mission Street, San Francisco, CA-94103, USA
> > <
> >
> https://maps.google.com/?q=1161+Mission+Street,+San+Francisco,+CA-94103,+USA&entry=gmail&source=g
> > >
> >
> > --
> > Data Artisans GmbH
> > Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
> >
>

[Becket Qin]

Thanks for the feedback, Shaoxuan.

I'll create a FLIP then. Good point about documentation. I'll put together
a doc on the website once we agreed upon the detail metric specifications.

Jiangjie (Becket) Qin

On Mon, Feb 11, 2019 at 4:10 PM Shaoxuan Wang <[hidden email]> wrote:

> Becket,
>
> Thanks for the proposal. I am much agree that we should only design the
> standard/common Flink-related connector metrics, while keep the original
> names for the ones provided by the connector itself. If everyone agrees to
> make these changes, we should do it asap, the earlier to make the change
> the less impact on the user's system. Maybe we can start to unify the
> metrics on the existing connectors, and then slowly introduce the new
> common/standard metrics if necessary.
>
> Since metric is also a kind of user interface, we should either open a
> Flip, or create a webpage to document the specifications, it's up to you.
> The point is that we want to create a good document, such that the future
> developers can easily find and follow these specifications.
>
> Regards,
> Shaoxuan
>
> On Fri, Feb 1, 2019 at 5:15 PM Becket Qin <[hidden email]> wrote:
>
> > Thanks for the feedback, Konstantin,
> >
> > Completely agree that exposing the original metrics from the clients of
> > other systems is a good practice and should be encouraged.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Fri, Feb 1, 2019 at 4:01 PM Konstantin Knauf <
> > [hidden email]>
> > wrote:
> >
> > > Hi Becket,
> > >
> > > I like the idea of providing a standard set of metrics, which sources
> can
> > > choose to implement/expose. In addition, I think, sources, like the
> Kafka
> > > or Kinesis Source, should continue to forward the original consumer
> > metrics
> > > under their original names, so that users familiar with Kafka/Kinesis
> can
> > > relate to them easily.
> > >
> > > Cheers,
> > >
> > > Konstantin
> > >
> > >
> > >
> > > On Fri, Feb 1, 2019 at 3:54 AM Becket Qin <[hidden email]>
> wrote:
> > >
> > > > Thanks for the connector metric url,  Chesnay :)
> > > >
> > > > @Robert, as you can see, the metrics from different connectors are
> > quite
> > > > different. And there are different names for similar metrics, which
> is
> > a
> > > > little frustrating when users want to do monitoring / alerting.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On Thu, Jan 31, 2019 at 6:41 PM Chesnay Schepler <[hidden email]
> >
> > > > wrote:
> > > >
> > > > > @Robert:
> > > > >
> > > > >
> > > >
> > >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#connectors
> > > > >
> > > > > On 31.01.2019 11:03, Robert Metzger wrote:
> > > > > > Hey Becket,
> > > > > > thanks a lot for your proposal!
> > > > > >
> > > > > > Do you have an overview over the current situation of the metrics
> > in
> > > > the
> > > > > > connectors?
> > > > > > Which connectors expose metrics at all?
> > > > > > Are they different?
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 8:44 AM Becket Qin <[hidden email]
> >
> > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> I was trying to add some metrics to Kafka connectors and
> realized
> > > that
> > > > > >> right now Flink does not have a common metric definition for the
> > > > > >> connectors. This complicates the monitoring and operation
> because
> > > the
> > > > > >> monitoring / alerts need to be set case by case.
> > > > > >>
> > > > > >> To address this issue, I would like to see if is possible to
> have
> > a
> > > > set
> > > > > of
> > > > > >> standardized common metrics for all sources and sinks. The
> > following
> > > > doc
> > > > > >> describes the proposal. Feedback is very welcome.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1q86bgj_3T6WFbSUoxLDJJXmUcBOUcvWfh2RZvHG-nPU/edit#
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Jiangjie (Becket) Qin
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > --
> > >
> > > Konstantin Knauf | Solution Architect
> > >
> > >
> > > --
> > >
> > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> > > Conference
> > >
> > > Stream Processing | Event Driven | Real Time
> > >
> > > --
> > >
> > > Data Artisans GmbH | Stresemannstr. 121A,10963 Berlin, Germany
> > > <
> > >
> >
> https://maps.google.com/?q=Stresemannstr.+121A,10963+Berlin,+Germany&entry=gmail&source=g
> > > >
> > > data Artisans, Inc. | 1161 Mission Street, San Francisco, CA-94103, USA
> > > <
> > >
> >
> https://maps.google.com/?q=1161+Mission+Street,+San+Francisco,+CA-94103,+USA&entry=gmail&source=g
> > > >
> > >
> > > --
> > > Data Artisans GmbH
> > > Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
> > >
> >
>