[jira] [Created] (FLINK-12161) Supports partial-final optimization for stream group aggregate

[Shang Yuanchun (Jira)]

godfrey he created FLINK-12161:
----------------------------------

             Summary:  Supports partial-final optimization for stream group aggregate
                 Key: FLINK-12161
                 URL: https://issues.apache.org/jira/browse/FLINK-12161
             Project: Flink
          Issue Type: New Feature
          Components: Table SQL / Planner
            Reporter: godfrey he
            Assignee: godfrey he


To resolve data-skew for distinct aggregates on stream, we introduce a rule named {{SplitAggregateRule}} which rewrites an aggregate query with distinct aggregations into an expanded double aggregations. The first aggregation compute the results in sub-partition(with bucket) and the results are combined by the second aggregation.

if two-stage aggregation is also enabled, we find that many plans have common pattern, looks like:

{code}
...
StreamExecGlobalGroupAggregate (final global agg)
+- StreamExecExchange
     +- StreamExecLocalGroupAggregate (final local agg)
          +- StreamExecGlobalGroupAggregate (partial global agg)
               +- ....
{code}

There is no exchange between the final local aggregate and the partial global aggregate, so they will be executed in a same JobVertex, and could share state. We introduce a rule named {{IncrementalAggregateRule}} to do that optimization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)