|
Hi all,
during a workflow, a data set may run empty, e.g., because of a join without matches. We're using FlinkTestBase and found out, that aggregate functions on empty data sets work fine in CLUSTER execution mode but cause a Null Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION mode. Here is the minimal example on 1.0-SNAPSHOT: https://gist.github.com/p3et/59a65bab11098dd11054 Are we doing something wrong, or is this a bug? Cheers, Andre -- ------------------------------------------- PhD Student University of Leipzig Department of Computer Science Database Research Group email: [hidden email] web: dbs.uni-leipzig.de ------------------------------------------- |
|
Hi,
What he meant was MultipleProgramsTestBase, not FlinkTestBase. I debugged this a bit. The NPE is thrown in https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java#L296 since current can be null if the input iterator is empty. In Cluster Execution, it is checked that the output of the previous function (e.g. Filter) is not empty in: https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllGroupReduceDriver.java#L144 which avoids going into AggregateOperator and getting a NPE. However, in Collection Mode, the execution is not grouped (don't know why, yet). In https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java#L207 the copied input data is handed over to the aggregate function which leads to the NPE. Checking inputDataCopy.size() > 0 before calling the aggregate solves the problem. If someone can confirm that this is not a more generic problem, I would open an issue and a PR. Best, Martin On 20.11.2015 18:41, André Petermann wrote: > Hi all, > > during a workflow, a data set may run empty, e.g., because of a join > without matches. > > We're using FlinkTestBase and found out, that aggregate functions on > empty data sets work fine in CLUSTER execution mode but cause a Null > Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION mode. > > Here is the minimal example on 1.0-SNAPSHOT: > https://gist.github.com/p3et/59a65bab11098dd11054 > > Are we doing something wrong, or is this a bug? > > Cheers, > Andre > |
|
Hi André, hi Martin,
This looks very much like a bug. Martin, I would be happy if you opened a JIRA issue. Thanks, Max On Sun, Nov 22, 2015 at 12:27 PM, Martin Junghanns <[hidden email]> wrote: > Hi, > > What he meant was MultipleProgramsTestBase, not FlinkTestBase. > > I debugged this a bit. > > The NPE is thrown in > > https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java#L296 > > since current can be null if the input iterator is empty. > > In Cluster Execution, it is checked that the output of the previous function > (e.g. Filter) is not empty in: > > https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllGroupReduceDriver.java#L144 > > which avoids going into AggregateOperator and getting a NPE. > > However, in Collection Mode, the execution is not grouped (don't know why, > yet). In > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java#L207 > > the copied input data is handed over to the aggregate function which leads > to the NPE. > > Checking inputDataCopy.size() > 0 before calling the aggregate solves the > problem. > > If someone can confirm that this is not a more generic problem, I would open > an issue and a PR. > > Best, > Martin > > > On 20.11.2015 18:41, André Petermann wrote: >> >> Hi all, >> >> during a workflow, a data set may run empty, e.g., because of a join >> without matches. >> >> We're using FlinkTestBase and found out, that aggregate functions on >> empty data sets work fine in CLUSTER execution mode but cause a Null >> Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION mode. >> >> Here is the minimal example on 1.0-SNAPSHOT: >> https://gist.github.com/p3et/59a65bab11098dd11054 >> >> Are we doing something wrong, or is this a bug? >> >> Cheers, >> Andre >> > |
|
Hi Max,
fixed in https://github.com/apache/flink/pull/1396 Best, Martin On 24.11.2015 13:46, Maximilian Michels wrote: > Hi André, hi Martin, > > This looks very much like a bug. Martin, I would be happy if you > opened a JIRA issue. > > Thanks, > Max > > On Sun, Nov 22, 2015 at 12:27 PM, Martin Junghanns > <[hidden email]> wrote: >> Hi, >> >> What he meant was MultipleProgramsTestBase, not FlinkTestBase. >> >> I debugged this a bit. >> >> The NPE is thrown in >> >> https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java#L296 >> >> since current can be null if the input iterator is empty. >> >> In Cluster Execution, it is checked that the output of the previous function >> (e.g. Filter) is not empty in: >> >> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllGroupReduceDriver.java#L144 >> >> which avoids going into AggregateOperator and getting a NPE. >> >> However, in Collection Mode, the execution is not grouped (don't know why, >> yet). In >> >> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java#L207 >> >> the copied input data is handed over to the aggregate function which leads >> to the NPE. >> >> Checking inputDataCopy.size() > 0 before calling the aggregate solves the >> problem. >> >> If someone can confirm that this is not a more generic problem, I would open >> an issue and a PR. >> >> Best, >> Martin >> >> >> On 20.11.2015 18:41, André Petermann wrote: >>> >>> Hi all, >>> >>> during a workflow, a data set may run empty, e.g., because of a join >>> without matches. >>> >>> We're using FlinkTestBase and found out, that aggregate functions on >>> empty data sets work fine in CLUSTER execution mode but cause a Null >>> Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION mode. >>> >>> Here is the minimal example on 1.0-SNAPSHOT: >>> https://gist.github.com/p3et/59a65bab11098dd11054 >>> >>> Are we doing something wrong, or is this a bug? >>> >>> Cheers, >>> Andre >>> >> |
|
Hi Martin,
Great. Thanks for the fix! Cheers, Max On Tue, Nov 24, 2015 at 7:40 PM, Martin Junghanns <[hidden email]> wrote: > Hi Max, > > fixed in https://github.com/apache/flink/pull/1396 > > Best, > Martin > > > On 24.11.2015 13:46, Maximilian Michels wrote: >> >> Hi André, hi Martin, >> >> This looks very much like a bug. Martin, I would be happy if you >> opened a JIRA issue. >> >> Thanks, >> Max >> >> On Sun, Nov 22, 2015 at 12:27 PM, Martin Junghanns >> <[hidden email]> wrote: >>> >>> Hi, >>> >>> What he meant was MultipleProgramsTestBase, not FlinkTestBase. >>> >>> I debugged this a bit. >>> >>> The NPE is thrown in >>> >>> >>> https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java#L296 >>> >>> since current can be null if the input iterator is empty. >>> >>> In Cluster Execution, it is checked that the output of the previous >>> function >>> (e.g. Filter) is not empty in: >>> >>> >>> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllGroupReduceDriver.java#L144 >>> >>> which avoids going into AggregateOperator and getting a NPE. >>> >>> However, in Collection Mode, the execution is not grouped (don't know >>> why, >>> yet). In >>> >>> >>> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java#L207 >>> >>> the copied input data is handed over to the aggregate function which >>> leads >>> to the NPE. >>> >>> Checking inputDataCopy.size() > 0 before calling the aggregate solves the >>> problem. >>> >>> If someone can confirm that this is not a more generic problem, I would >>> open >>> an issue and a PR. >>> >>> Best, >>> Martin >>> >>> >>> On 20.11.2015 18:41, André Petermann wrote: >>>> >>>> >>>> Hi all, >>>> >>>> during a workflow, a data set may run empty, e.g., because of a join >>>> without matches. >>>> >>>> We're using FlinkTestBase and found out, that aggregate functions on >>>> empty data sets work fine in CLUSTER execution mode but cause a Null >>>> Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION >>>> mode. >>>> >>>> Here is the minimal example on 1.0-SNAPSHOT: >>>> https://gist.github.com/p3et/59a65bab11098dd11054 >>>> >>>> Are we doing something wrong, or is this a bug? >>>> >>>> Cheers, >>>> Andre >>>> >>> > |
|
In reply to this post by André Petermann
ull pointer errors that appear only in COLLECTION mode can be tricky to diagnose, especially when tests behave differently across execution environments. Checking how data is initialized and handled in each mode can help narrow down the cause, while tools like https://null-brawl.io/ offer a different kind of gameplay experience for those looking for a break from debugging.
|
|
In reply to this post by André Petermann
When troubleshooting test failures that occur only in a specific execution mode, it helps to compare how different configurations handle data collection and object initialization, as the issue may be related to mode specific behavior rather than the test logic itself. Similar problems can sometimes be easier to investigate by reviewing application requirements and checking the necessary setup details before proceeding, and anyone planning a trip can also visit website to understand the online Saudi visa application process.
|
|
In reply to this post by André Petermann
Debugging issues that appear only in specific operating modes requires careful testing and a systematic approach to identify the underlying cause. A similar attention to detail matters when diagnosing appliance faults, where checking individual components can help prevent unnecessary replacements. Having access to Al Haram Electrical Appliances Repairs Shop can be useful when practical appliance troubleshooting and reliable repair support are needed.
|
|
In reply to this post by André Petermann
When troubleshooting Apache Flink tests, especially issues that appear only in COLLECTION mode, it is important to check how data is processed and how different execution environments handle object references. Proper business planning also involves understanding complex workflows and resolving unexpected issues, and EBFS offers useful insights into structured business processes that require careful coordination.
|
|
In reply to this post by André Petermann
When dealing with complex technical issues like Null Pointer Exceptions in Apache Flink, having the right guidance makes a real difference. The same applies to students preparing for demanding exams, where home tuition in dha lahore can provide subject specialists for O-Level, A-Level, Mathematics, Physics, Chemistry, and other subjects, with flexible home or online lessons to support individual learning needs.
|
|
In reply to this post by André Petermann
When investigating Flink test failures that occur only in COLLECTION mode, isolating execution-specific behavior and checking how test data is handled can help identify the cause of a Null Pointer Exception. Similarly, having the right customization options makes a difference in gaming, where a unique nickname can give your profile more personality. Players looking for stylish fonts, decorative symbols, and creative Free Fire names can explore https://sekilli-nick.com/free-fire-nick/ to find a nickname style that matches their gaming identity.
|
|
In reply to this post by André Petermann
Collection mode can sometimes expose differences in how test data is initialized or handled, particularly when null values are encountered during execution. Similar issues in other systems often require checking the surrounding configuration and memory or data handling before assuming the test logic itself is faulty. It may help to check here for additional troubleshooting context when investigating unexpected behavior.
|
| Free forum by Nabble | Edit this page |
