flink release-1.8.0 Flink-avro unit tests failed

[Ethan Li]

Hello,

Not sure if anyone encountered this issue before.  I tried to run “mvn clean install” on flink release-1.8, but some unit tests in flink-arvo module failed:


[ERROR] Tests run: 12, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 4.81 s <<< FAILURE! - in org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest
[ERROR] testSimpleAvroRead[Execution mode = CLUSTER](org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest)  Time elapsed: 0.438 s  <<< FAILURE!
java.lang.AssertionError:
Different elements in arrays: expected 2 elements and received 2
files: [/tmp/junit5386344396421857812/junit6023978980792200274.tmp/4, /tmp/junit5386344396421857812/junit6023978980792200274.tmp/2, /tmp/junit5386344396421857812/junit6023978980792200274.tmp/1, /tmp/junit5386344396421857812/junit6023978980792200274.tmp/3]
 expected: [{"name": "Alyssa", "favorite_number": 256, "favorite_color": null, "type_long_test": null, "type_double_test": 123.45, "type_null_test": null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT 2"], "type_array_boolean": [true, false], "type_nullable_array": null, "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7, -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color": "blue", "type_long_test": 1337, "type_double_test": 1.337, "type_null_test": null, "type_bool_test": false, "type_array_string": [], "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED", "type_map": {}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7, -48]}]
 received: [{"name": "Alyssa", "favorite_number": 256, "favorite_color": null, "type_long_test": null, "type_double_test": 123.45, "type_null_test": null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT 2"], "type_array_boolean": [true, false], "type_nullable_array": null, "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7, -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color": "blue", "type_long_test": 1337, "type_double_test": 1.337, "type_null_test": null, "type_bool_test": false, "type_array_string": [], "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED", "type_map": {}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7, -48]}]
        at org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest.after(AvroTypeExtractionTest.java:76)



Comparing “expected” with “received”, there is really some question mark difference.

For example, in “expected’, it’s

"type_decimal_bytes": {"bytes": "\u0007?”}

While in “received”, it’s

"type_decimal_bytes": {"bytes": "\u0007??"}


I would really appreciate it if anyone could shed some light on this. Thanks

Ethan

[Ethan Li]

It’s probably the encoding problem. The environment I ran the unit tests on uses ANSI_X3.4-1968
 
It looks like we have to use "en_US.UTF-8"

> On Aug 19, 2019, at 1:44 PM, Ethan Li <[hidden email]> wrote:
>
> Hello,
>
> Not sure if anyone encountered this issue before.  I tried to run “mvn clean install” on flink release-1.8, but some unit tests in flink-arvo module failed:
>
>
> [ERROR] Tests run: 12, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 4.81 s <<< FAILURE! - in org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest
> [ERROR] testSimpleAvroRead[Execution mode = CLUSTER](org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest)  Time elapsed: 0.438 s  <<< FAILURE!
> java.lang.AssertionError:
> Different elements in arrays: expected 2 elements and received 2
> files: [/tmp/junit5386344396421857812/junit6023978980792200274.tmp/4, /tmp/junit5386344396421857812/junit6023978980792200274.tmp/2, /tmp/junit5386344396421857812/junit6023978980792200274.tmp/1, /tmp/junit5386344396421857812/junit6023978980792200274.tmp/3]
>  expected: [{"name": "Alyssa", "favorite_number": 256, "favorite_color": null, "type_long_test": null, "type_double_test": 123.45, "type_null_test": null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT 2"], "type_array_boolean": [true, false], "type_nullable_array": null, "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7, -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color": "blue", "type_long_test": 1337, "type_double_test": 1.337, "type_null_test": null, "type_bool_test": false, "type_array_string": [], "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED", "type_map": {}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7, -48]}]
>  received: [{"name": "Alyssa", "favorite_number": 256, "favorite_color": null, "type_long_test": null, "type_double_test": 123.45, "type_null_test": null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT 2"], "type_array_boolean": [true, false], "type_nullable_array": null, "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7, -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color": "blue", "type_long_test": 1337, "type_double_test": 1.337, "type_null_test": null, "type_bool_test": false, "type_array_string": [], "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED", "type_map": {}, "type_fixed": null, "type_union": null, "type_nested": {"num": 239, "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1 6XE"}, "type_bytes": {"bytes": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}, "type_date": 2014-03-01, "type_time_millis": 12:12:12.000, "type_time_micros": 123456, "type_timestamp_millis": 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456, "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7, -48]}]
> at org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest.after(AvroTypeExtractionTest.java:76)
>
>
>
> Comparing “expected” with “received”, there is really some question mark difference.
>
> For example, in “expected’, it’s
>
> "type_decimal_bytes": {"bytes": "\u0007?”}
>
> While in “received”, it’s
>
> "type_decimal_bytes": {"bytes": "\u0007??"}
>
>
> I would really appreciate it if anyone could shed some light on this. Thanks
>
> Ethan

[Stephan Ewen]

Thanks, looks like you diagnosed it correctly. environment specific
encoding settings.

Could you open a ticket (maybe a PR) to set the encoding and make the test
stable across environments?

On Mon, Aug 19, 2019 at 9:46 PM Ethan Li <[hidden email]> wrote:

> It’s probably the encoding problem. The environment I ran the unit tests
> on uses ANSI_X3.4-1968
>
> It looks like we have to use "en_US.UTF-8"
>
>
> > On Aug 19, 2019, at 1:44 PM, Ethan Li <[hidden email]> wrote:
> >
> > Hello,
> >
> > Not sure if anyone encountered this issue before.  I tried to run “mvn
> clean install” on flink release-1.8, but some unit tests in flink-arvo
> module failed:
> >
> >
> > [ERROR] Tests run: 12, Failures: 4, Errors: 0, Skipped: 0, Time elapsed:
> 4.81 s <<< FAILURE! - in
> org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest
> > [ERROR] testSimpleAvroRead[Execution mode =
> CLUSTER](org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest)
> Time elapsed: 0.438 s  <<< FAILURE!
> > java.lang.AssertionError:
> > Different elements in arrays: expected 2 elements and received 2
> > files: [/tmp/junit5386344396421857812/junit6023978980792200274.tmp/4,
> /tmp/junit5386344396421857812/junit6023978980792200274.tmp/2,
> /tmp/junit5386344396421857812/junit6023978980792200274.tmp/1,
> /tmp/junit5386344396421857812/junit6023978980792200274.tmp/3]
> >  expected: [{"name": "Alyssa", "favorite_number": 256, "favorite_color":
> null, "type_long_test": null, "type_double_test": 123.45, "type_null_test":
> null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT
> 2"], "type_array_boolean": [true, false], "type_nullable_array": null,
> "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456},
> "type_fixed": null, "type_union": null, "type_nested": {"num": 239,
> "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1
> 6XE"}, "type_bytes": {"bytes":
> "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7,
> -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color":
> "blue", "type_long_test": 1337, "type_double_test": 1.337,
> "type_null_test": null, "type_bool_test": false, "type_array_string": [],
> "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED",
> "type_map": {}, "type_fixed": null, "type_union": null, "type_nested":
> {"num": 239, "street": "Baker Street", "city": "London", "state": "London",
> "zip": "NW1 6XE"}, "type_bytes": {"bytes":
> "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7, -48]}]
> >  received: [{"name": "Alyssa", "favorite_number": 256, "favorite_color":
> null, "type_long_test": null, "type_double_test": 123.45, "type_null_test":
> null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT
> 2"], "type_array_boolean": [true, false], "type_nullable_array": null,
> "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456},
> "type_fixed": null, "type_union": null, "type_nested": {"num": 239,
> "street": "Baker Street", "city": "London", "state": "London", "zip": "NW1
> 6XE"}, "type_bytes": {"bytes":
> "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7,
> -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color":
> "blue", "type_long_test": 1337, "type_double_test": 1.337,
> "type_null_test": null, "type_bool_test": false, "type_array_string": [],
> "type_array_boolean": [], "type_nullable_array": null, "type_enum": "RED",
> "type_map": {}, "type_fixed": null, "type_union": null, "type_nested":
> {"num": 239, "street": "Baker Street", "city": "London", "state": "London",
> "zip": "NW1 6XE"}, "type_bytes": {"bytes":
> "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> "type_time_micros": 123456, "type_timestamp_millis":
> 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7,
> -48]}]
> >       at
> org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest.after(AvroTypeExtractionTest.java:76)
> >
> >
> >
> > Comparing “expected” with “received”, there is really some question mark
> difference.
> >
> > For example, in “expected’, it’s
> >
> > "type_decimal_bytes": {"bytes": "\u0007?”}
> >
> > While in “received”, it’s
> >
> > "type_decimal_bytes": {"bytes": "\u0007??"}
> >
> >
> > I would really appreciate it if anyone could shed some light on this.
> Thanks
> >
> > Ethan
>
>

[Ethan Li]

I filed a jira https://issues.apache.org/jira/browse/FLINK-13807 for this.
Not sure if I am able to get to it in the near future so I didn't assign to
myself. Anyone feel free to pick it up.

I changed my environment to pass this for now.

On Tue, Aug 20, 2019 at 4:11 AM Stephan Ewen <[hidden email]> wrote:

> Thanks, looks like you diagnosed it correctly. environment specific
> encoding settings.
>
> Could you open a ticket (maybe a PR) to set the encoding and make the test
> stable across environments?
>
> On Mon, Aug 19, 2019 at 9:46 PM Ethan Li <[hidden email]>
> wrote:
>
> > It’s probably the encoding problem. The environment I ran the unit tests
> > on uses ANSI_X3.4-1968
> >
> > It looks like we have to use "en_US.UTF-8"
> >
> >
> > > On Aug 19, 2019, at 1:44 PM, Ethan Li <[hidden email]>
> wrote:
> > >
> > > Hello,
> > >
> > > Not sure if anyone encountered this issue before.  I tried to run “mvn
> > clean install” on flink release-1.8, but some unit tests in flink-arvo
> > module failed:
> > >
> > >
> > > [ERROR] Tests run: 12, Failures: 4, Errors: 0, Skipped: 0, Time
> elapsed:
> > 4.81 s <<< FAILURE! - in
> > org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest
> > > [ERROR] testSimpleAvroRead[Execution mode =
> > CLUSTER](org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest)
> > Time elapsed: 0.438 s  <<< FAILURE!
> > > java.lang.AssertionError:
> > > Different elements in arrays: expected 2 elements and received 2
> > > files: [/tmp/junit5386344396421857812/junit6023978980792200274.tmp/4,
> > /tmp/junit5386344396421857812/junit6023978980792200274.tmp/2,
> > /tmp/junit5386344396421857812/junit6023978980792200274.tmp/1,
> > /tmp/junit5386344396421857812/junit6023978980792200274.tmp/3]
> > >  expected: [{"name": "Alyssa", "favorite_number": 256,
> "favorite_color":
> > null, "type_long_test": null, "type_double_test": 123.45,
> "type_null_test":
> > null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT
> > 2"], "type_array_boolean": [true, false], "type_nullable_array": null,
> > "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456},
> > "type_fixed": null, "type_union": null, "type_nested": {"num": 239,
> > "street": "Baker Street", "city": "London", "state": "London", "zip":
> "NW1
> > 6XE"}, "type_bytes": {"bytes":
> > "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> > "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> > "type_time_micros": 123456, "type_timestamp_millis":
> > 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> > "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7,
> > -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color":
> > "blue", "type_long_test": 1337, "type_double_test": 1.337,
> > "type_null_test": null, "type_bool_test": false, "type_array_string": [],
> > "type_array_boolean": [], "type_nullable_array": null, "type_enum":
> "RED",
> > "type_map": {}, "type_fixed": null, "type_union": null, "type_nested":
> > {"num": 239, "street": "Baker Street", "city": "London", "state":
> "London",
> > "zip": "NW1 6XE"}, "type_bytes": {"bytes":
> > "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> > "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> > "type_time_micros": 123456, "type_timestamp_millis":
> > 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> > "type_decimal_bytes": {"bytes": "\u0007?"}, "type_decimal_fixed": [7,
> -48]}]
> > >  received: [{"name": "Alyssa", "favorite_number": 256,
> "favorite_color":
> > null, "type_long_test": null, "type_double_test": 123.45,
> "type_null_test":
> > null, "type_bool_test": true, "type_array_string": ["ELEMENT 1", "ELEMENT
> > 2"], "type_array_boolean": [true, false], "type_nullable_array": null,
> > "type_enum": "GREEN", "type_map": {"KEY 2": 17554, "KEY 1": 8546456},
> > "type_fixed": null, "type_union": null, "type_nested": {"num": 239,
> > "street": "Baker Street", "city": "London", "state": "London", "zip":
> "NW1
> > 6XE"}, "type_bytes": {"bytes":
> > "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> > "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> > "type_time_micros": 123456, "type_timestamp_millis":
> > 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> > "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7,
> > -48]}, {"name": "Charlie", "favorite_number": null, "favorite_color":
> > "blue", "type_long_test": 1337, "type_double_test": 1.337,
> > "type_null_test": null, "type_bool_test": false, "type_array_string": [],
> > "type_array_boolean": [], "type_nullable_array": null, "type_enum":
> "RED",
> > "type_map": {}, "type_fixed": null, "type_union": null, "type_nested":
> > {"num": 239, "street": "Baker Street", "city": "London", "state":
> "London",
> > "zip": "NW1 6XE"}, "type_bytes": {"bytes":
> > "\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},
> > "type_date": 2014-03-01, "type_time_millis": 12:12:12.000,
> > "type_time_micros": 123456, "type_timestamp_millis":
> > 2014-03-01T12:12:12.321Z, "type_timestamp_micros": 123456,
> > "type_decimal_bytes": {"bytes": "\u0007??"}, "type_decimal_fixed": [7,
> > -48]}]
> > >       at
> >
> org.apache.flink.formats.avro.typeutils.AvroTypeExtractionTest.after(AvroTypeExtractionTest.java:76)
> > >
> > >
> > >
> > > Comparing “expected” with “received”, there is really some question
> mark
> > difference.
> > >
> > > For example, in “expected’, it’s
> > >
> > > "type_decimal_bytes": {"bytes": "\u0007?”}
> > >
> > > While in “received”, it’s
> > >
> > > "type_decimal_bytes": {"bytes": "\u0007??"}
> > >
> > >
> > > I would really appreciate it if anyone could shed some light on this.
> > Thanks
> > >
> > > Ethan
> >
> >
>