Nico Kruber created FLINK-12983:
-----------------------------------
Summary: Replace descriptive histogram's storage back-end
Key: FLINK-12983
URL:
https://issues.apache.org/jira/browse/FLINK-12983 Project: Flink
Issue Type: Sub-task
Components: Runtime / Metrics
Reporter: Nico Kruber
Assignee: Nico Kruber
{{DescriptiveStatistics}} relies on their {{ResizableDoubleArray}} for storing double values for their histograms. However, this is constantly resizing an internal array and seems to have quite some overhead.
Additionally, we're not using {{SynchronizedDescriptiveStatistics}} which, according to its docs, we should. Currently, we seem to be somewhat safe because {{ResizableDoubleArray}} has some synchronized parts but these are scheduled to go away with commons.math version 4.
Internal tests with the current implementation, one based on a linear array of twice the histogram size (and moving values back to the start once the window reaches the end), and one using a circular array (wrapping around with flexible start position) has shown these numbers using the optimised code from FLINK-10236, FLINK-12981, and FLINK-12982:
# only adding values to the histogram
{code}
Benchmark Mode Cnt Score Error Units
HistogramBenchmarks.dropwizardHistogramAdd thrpt 30 47985.359 ± 25.847 ops/ms
HistogramBenchmarks.descriptiveHistogramAdd thrpt 30 70158.792 ± 276.858 ops/ms
--- with FLINK-10236, FLINK-12981, and FLINK-12982 ---
HistogramBenchmarks.descriptiveHistogramAdd thrpt 30 75303.040 ± 475.355 ops/ms
HistogramBenchmarks.histogramCircularArrayAdd thrpt 30 790123.475 ± 48420.672 ops/ms
HistogramBenchmarks.histogramLinearArrayAdd thrpt 30 385126.074 ± 3038.773 ops/ms
{code}
# after adding each value, also retrieving a common set of metrics:
{code}
Benchmark Mode Cnt Score Error Units
HistogramBenchmarks.dropwizardHistogram thrpt 30 400.274 ± 4.930 ops/ms
HistogramBenchmarks.descriptiveHistogram thrpt 30 124.533 ± 1.060 ops/ms
--- with FLINK-10236, FLINK-12981, and FLINK-12982 ---
HistogramBenchmarks.descriptiveHistogram thrpt 30 251.895 ± 1.809 ops/ms
HistogramBenchmarks.histogramCircularArray thrpt 30 298.881 ± 10.027 ops/ms
HistogramBenchmarks.histogramLinearArray thrpt 30 234.380 ± 5.014 ops/ms
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Locked
