Re: Unit testing for ProcessAllWindowFunction

[Diogo Araújo]

Even though your answer was helpful, I am testing my flink pipeline in scala not in java.
Is there any Scala example you can provide me?
Thanks in advance

Diogo Araújo | Rockstar Developer
[hidden email]
[phone]

Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
www.criticaltechworks.com <https://www.criticaltechworks.com/>
  <https://www.instagram.com/criticaltechworks/>  <https://www.linkedin.com/company/criticaltechworks/>  <https://twitter.com/ctechworks> <https://www.facebook.com/CriticalTechWorks/>
 

On 01/11/2019, 03:38, "vino yang" <[hidden email]> wrote:

    Hi Diogo,
   
    In order to test ProcessAllWindowFunction, you need to do a little more.
   
    You can refer to Flink's own test code.[1]
   
    In short:
   
    1) Write a Flink Streaming job that uses your UDF (here is
    ProcessAllWindowFunction) and return a DataStream.
    2) Get OneInputTransformation through DataStream, and then get
    OneInputStreamOperator;
    3) Convert OneInputStreamOperator to WindowOperator;
    4) Use test harness, please refer here[2]
   
    Best,
    Vino
   
    [1]:
    https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/AllWindowTranslationTest.java#L732
    [2]:
    https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/AllWindowTranslationTest.java#L1405
   
    Diogo Araújo <[hidden email]> 于2019年11月1日周五 上午1:31写道：
   
    > Good afternoon,
    >
    >
    >
    > After Reading the official flink testing documentation (
    > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html
    > )
    >
    > I was able to develop tests for a ProcessFunction, using a Test Harness,
    > something like this:
    >
    >
    >
    > *pendingPartitionBuilder *= new PendingPartitionBuilder(":::some_name", ""
    > )
    >
    > *testHarness *=
    >   new OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,
    > PendingPartition](
    >     new ProcessOperator[StaticAdequacyTilePublishedData,PendingPartition](
    > *pendingPartitionBuilder*)
    >   )
    >
    > *testHarness*.open()
    >
    >
    >
    >
    > now, I’m trying to do the same for a ProcessAllWindowFunction.
    >
    > First I realized I can’t use TestHarness for  ProcessAllWindowFunction,
    > because it doesn’t have a processElement method. In this case, what unit
    > test strategy should I follow?
    >
    >
    >
    >
    >
    > *Diogo Araújo* | Rockstar Developer
    > [hidden email]
    > [phone]
    > [image: Critical TechWorks]
    > Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
    > www.criticaltechworks.com
    > [image: Critical TechWorks @ Instagram]
    > <https://www.instagram.com/criticaltechworks/>[image: Critical TechWorks
    > @ LinkedIn]  <https://www.linkedin.com/company/criticaltechworks/>[image:
    > Critical TechWorks @ Twitter]  <https://twitter.com/ctechworks>[image:
    > Critical TechWorks @ Facebook]
    > <https://www.facebook.com/CriticalTechWorks/>
    >
    >
    >
    >
    >
    > *From: *Diogo Araújo <[hidden email]>
    > *Date: *Thursday, 31 October 2019 at 16:55
    > *To: *"[hidden email]" <[hidden email]>
    > *Subject: *Unit testing for ProcessAllWindowFunction
    >
    >
    >
    > Good afternoon,
    >
    >
    >
    > After Reading the official flink testing documentation (
    > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html
    > )
    >
    > I was able to develop tests for a ProcessFunction, using a Test Harness,
    > something like this:
    >
    >
    >
    > *pendingPartitionBuilder *= new PendingPartitionBuilder(":::some_name", ""
    > )
    >
    > *testHarness *=
    >   new OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,
    > PendingPartition](
    >     new ProcessOperator[StaticAdequacyTilePublishedData,
    > PendingPartition](*pendingPartitionBuilder*)
    >   )
    >
    > *testHarness*.open()
    >
    >
    >
    >
    > now, I’m trying to do the same for a ProcessAllWindowFunction.
    >
    > First I realized I can’t use TestHarness for  ProcessAllWindowFunction,
    > because it doesn’t have a processElement method. In this case, what unit
    > test strategy should I follow?
    >
    > *Diogo Araújo* | Rockstar Developer
    > [hidden email]
    > [phone]
    > [image: Critical TechWorks]
    > Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
    > www.criticaltechworks.com
    > [image: Critical TechWorks @ Instagram]
    > <https://www.instagram.com/criticaltechworks/>[image: Critical TechWorks
    > @ LinkedIn]  <https://www.linkedin.com/company/criticaltechworks/>[image:
    > Critical TechWorks @ Twitter]  <https://twitter.com/ctechworks>[image:
    > Critical TechWorks @ Facebook]
    > <https://www.facebook.com/CriticalTechWorks/>
    >
    >
    >
   

[vino yang]

Hi Diogo,

There is also a Scala version about AllWindowTranslationTest[1] which
contains the example code snippet.

[1]:
https://github.com/apache/flink/blob/master/flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/AllWindowTranslationTest.scala#L1644

Best,
Vino

Diogo Araújo <[hidden email]> 于2019年11月4日周一 下午7:10写道：

> Even though your answer was helpful, I am testing my flink pipeline in
> scala not in java.
> Is there any Scala example you can provide me?
> Thanks in advance
>
> Diogo Araújo | Rockstar Developer
> [hidden email]
> [phone]
>
> Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
> www.criticaltechworks.com <https://www.criticaltechworks.com/>
>   <https://www.instagram.com/criticaltechworks/>  <
> https://www.linkedin.com/company/criticaltechworks/>  <
> https://twitter.com/ctechworks> <
> https://www.facebook.com/CriticalTechWorks/>
>
>
> On 01/11/2019, 03:38, "vino yang" <[hidden email]> wrote:
>
>     Hi Diogo,
>
>     In order to test ProcessAllWindowFunction, you need to do a little
> more.
>
>     You can refer to Flink's own test code.[1]
>
>     In short:
>
>     1) Write a Flink Streaming job that uses your UDF (here is
>     ProcessAllWindowFunction) and return a DataStream.
>     2) Get OneInputTransformation through DataStream, and then get
>     OneInputStreamOperator;
>     3) Convert OneInputStreamOperator to WindowOperator;
>     4) Use test harness, please refer here[2]
>
>     Best,
>     Vino
>
>     [1]:
>
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/AllWindowTranslationTest.java#L732
>     [2]:
>
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/AllWindowTranslationTest.java#L1405
>
>     Diogo Araújo <[hidden email]> 于2019年11月1日周五
> 上午1:31写道：
>
>     > Good afternoon,
>     >
>     >
>     >
>     > After Reading the official flink testing documentation (
>     >
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html
>     > )
>     >
>     > I was able to develop tests for a ProcessFunction, using a Test
> Harness,
>     > something like this:
>     >
>     >
>     >
>     > *pendingPartitionBuilder *= new
> PendingPartitionBuilder(":::some_name", ""
>     > )
>     >
>     > *testHarness *=
>     >   new
> OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,
>     > PendingPartition](
>     >     new
> ProcessOperator[StaticAdequacyTilePublishedData,PendingPartition](
>     > *pendingPartitionBuilder*)
>     >   )
>     >
>     > *testHarness*.open()
>     >
>     >
>     >
>     >
>     > now, I’m trying to do the same for a ProcessAllWindowFunction.
>     >
>     > First I realized I can’t use TestHarness for
> ProcessAllWindowFunction,
>     > because it doesn’t have a processElement method. In this case, what
> unit
>     > test strategy should I follow?
>     >
>     >
>     >
>     >
>     >
>     > *Diogo Araújo* | Rockstar Developer
>     > [hidden email]
>     > [phone]
>     > [image: Critical TechWorks]
>     > Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
>     > www.criticaltechworks.com
>     > [image: Critical TechWorks @ Instagram]
>     > <https://www.instagram.com/criticaltechworks/>[image: Critical
> TechWorks
>     > @ LinkedIn]  <https://www.linkedin.com/company/criticaltechworks/
> >[image:
>     > Critical TechWorks @ Twitter]  <https://twitter.com/ctechworks
> >[image:
>     > Critical TechWorks @ Facebook]
>     > <https://www.facebook.com/CriticalTechWorks/>
>     >
>     >
>     >
>     >
>     >
>     > *From: *Diogo Araújo <[hidden email]>
>     > *Date: *Thursday, 31 October 2019 at 16:55
>     > *To: *"[hidden email]" <[hidden email]>
>     > *Subject: *Unit testing for ProcessAllWindowFunction
>     >
>     >
>     >
>     > Good afternoon,
>     >
>     >
>     >
>     > After Reading the official flink testing documentation (
>     >
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/testing.html
>     > )
>     >
>     > I was able to develop tests for a ProcessFunction, using a Test
> Harness,
>     > something like this:
>     >
>     >
>     >
>     > *pendingPartitionBuilder *= new
> PendingPartitionBuilder(":::some_name", ""
>     > )
>     >
>     > *testHarness *=
>     >   new
> OneInputStreamOperatorTestHarness[StaticAdequacyTilePublishedData,
>     > PendingPartition](
>     >     new ProcessOperator[StaticAdequacyTilePublishedData,
>     > PendingPartition](*pendingPartitionBuilder*)
>     >   )
>     >
>     > *testHarness*.open()
>     >
>     >
>     >
>     >
>     > now, I’m trying to do the same for a ProcessAllWindowFunction.
>     >
>     > First I realized I can’t use TestHarness for
> ProcessAllWindowFunction,
>     > because it doesn’t have a processElement method. In this case, what
> unit
>     > test strategy should I follow?
>     >
>     > *Diogo Araújo* | Rockstar Developer
>     > [hidden email]
>     > [phone]
>     > [image: Critical TechWorks]
>     > Rua do Campo Alegre, nº 17, piso 0 | 4150-177 Porto
>     > www.criticaltechworks.com
>     > [image: Critical TechWorks @ Instagram]
>     > <https://www.instagram.com/criticaltechworks/>[image: Critical
> TechWorks
>     > @ LinkedIn]  <https://www.linkedin.com/company/criticaltechworks/
> >[image:
>     > Critical TechWorks @ Twitter]  <https://twitter.com/ctechworks
> >[image:
>     > Critical TechWorks @ Facebook]
>     > <https://www.facebook.com/CriticalTechWorks/>
>     >
>     >
>     >
>
>
>