FW: Unit testing for ProcessAllWindowFunction

[Diogo Araújo]

Good afternoon, 

 

After Reading the official flink testing documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html)

I was able to develop tests for a ProcessFunction, using a Test Harness, something like this: 

pendingPartitionBuilder = new PendingPartitionBuilder(":::some_name", "")

testHarness =
  new OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,PendingPartition](
    new ProcessOperator[StaticAdequacyTilePublishedData,PendingPartition](pendingPartitionBuilder)
  )

testHarness.open()

now, I’m trying to do the same for a ProcessAllWindowFunction.

First I realized I can’t use TestHarness for  ProcessAllWindowFunction, because it doesn’t have a processElement method. In this case, what unit test strategy should I follow?

 

 

Diogo Araújo | Rockstar Developer
[hidden email]
[phone]

Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
www.criticaltechworks.com
   

 

 

From: Diogo Araújo <[hidden email]>
Date: Thursday, 31 October 2019 at 16:55
To: "[hidden email]" <[hidden email]>
Subject: Unit testing for ProcessAllWindowFunction

 

Good afternoon,

 

After Reading the official flink testing documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html)

I was able to develop tests for a ProcessFunction, using a Test Harness, something like this:

pendingPartitionBuilder = new PendingPartitionBuilder(":::some_name", "")

testHarness =
  new OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData, PendingPartition](
    new ProcessOperator[StaticAdequacyTilePublishedData, PendingPartition](pendingPartitionBuilder)
  )

testHarness.open()

now, I’m trying to do the same for a ProcessAllWindowFunction.

First I realized I can’t use TestHarness for  ProcessAllWindowFunction, because it doesn’t have a processElement method. In this case, what unit test strategy should I follow?

Diogo Araújo | Rockstar Developer
[hidden email]
[phone]

Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
www.criticaltechworks.com
   

 

[vino yang]

Hi Diogo,

In order to test ProcessAllWindowFunction, you need to do a little more.

You can refer to Flink's own test code.[1]

In short:

1) Write a Flink Streaming job that uses your UDF (here is
ProcessAllWindowFunction) and return a DataStream.
2) Get OneInputTransformation through DataStream, and then get
OneInputStreamOperator;
3) Convert OneInputStreamOperator to WindowOperator;
4) Use test harness, please refer here[2]

Best,
Vino

[1]:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/AllWindowTranslationTest.java#L732
[2]:
https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/AllWindowTranslationTest.java#L1405

Diogo Araújo <[hidden email]> 于2019年11月1日周五 上午1:31写道：

> Good afternoon,
>
>
>
> After Reading the official flink testing documentation (
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html
> )
>
> I was able to develop tests for a ProcessFunction, using a Test Harness,
> something like this:
>
>
>
> *pendingPartitionBuilder *= new PendingPartitionBuilder(":::some_name", ""
> )
>
> *testHarness *=
>   new OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,
> PendingPartition](
>     new ProcessOperator[StaticAdequacyTilePublishedData,PendingPartition](
> *pendingPartitionBuilder*)
>   )
>
> *testHarness*.open()
>
>
>
>
> now, I’m trying to do the same for a ProcessAllWindowFunction.
>
> First I realized I can’t use TestHarness for  ProcessAllWindowFunction,
> because it doesn’t have a processElement method. In this case, what unit
> test strategy should I follow?
>
>
>
>
>
> *Diogo Araújo* | Rockstar Developer
> [hidden email]
> [phone]
> [image: Critical TechWorks]
> Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
> www.criticaltechworks.com
> [image: Critical TechWorks @ Instagram]
> <https://www.instagram.com/criticaltechworks/>[image: Critical TechWorks
> @ LinkedIn]  <https://www.linkedin.com/company/criticaltechworks/>[image:
> Critical TechWorks @ Twitter]  <https://twitter.com/ctechworks>[image:
> Critical TechWorks @ Facebook]
> <https://www.facebook.com/CriticalTechWorks/>
>
>
>
>
>
> *From: *Diogo Araújo <[hidden email]>
> *Date: *Thursday, 31 October 2019 at 16:55
> *To: *"[hidden email]" <[hidden email]>
> *Subject: *Unit testing for ProcessAllWindowFunction
>
>
>
> Good afternoon,
>
>
>
> After Reading the official flink testing documentation (
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html
> )
>
> I was able to develop tests for a ProcessFunction, using a Test Harness,
> something like this:
>
>
>
> *pendingPartitionBuilder *= new PendingPartitionBuilder(":::some_name", ""
> )
>
> *testHarness *=
>   new OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,
> PendingPartition](
>     new ProcessOperator[StaticAdequacyTilePublishedData,
> PendingPartition](*pendingPartitionBuilder*)
>   )
>
> *testHarness*.open()
>
>
>
>
> now, I’m trying to do the same for a ProcessAllWindowFunction.
>
> First I realized I can’t use TestHarness for  ProcessAllWindowFunction,
> because it doesn’t have a processElement method. In this case, what unit
> test strategy should I follow?
>
> *Diogo Araújo* | Rockstar Developer
> [hidden email]
> [phone]
> [image: Critical TechWorks]
> Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
> www.criticaltechworks.com
> [image: Critical TechWorks @ Instagram]
> <https://www.instagram.com/criticaltechworks/>[image: Critical TechWorks
> @ LinkedIn]  <https://www.linkedin.com/company/criticaltechworks/>[image:
> Critical TechWorks @ Twitter]  <https://twitter.com/ctechworks>[image:
> Critical TechWorks @ Facebook]
> <https://www.facebook.com/CriticalTechWorks/>
>
>
>