Experiment: Use native php tests

[MaartenStaa]

@czosel I had an idea the other day. A decent number of the bug reports on this project are due to small correctness issues. So I thought, is there a way to get ahead of the reports, and find them ourselves?

Well, the PHP project itself has many tests (.phpt files), so I set up this experiment. I pull in the php-src repository as a submodule, and made a script that generates Jest unit tests from the PHP tests, marking which should fail and which not. This branch and PR are the result.

As you can see in the commits, I've already tackled several correctness issues, but there are still plenty of failing tests. I'm jumping the gun and opening the PR to get early feedback for the idea and approach.

List of still failing tests:

• Modifying a readonly property (test/php-src/Zend-tests-readonly_props-readonly_modification.test.js)
• Attributes expose and verify target and repeatable data. (test/php-src/Zend-tests-attributes-020_userland_attribute_validation.test.js)
• Test semi-reserved words as class constants (test/php-src/Zend-tests-grammar-semi_reserved_005.test.js)
• Reserved keywords in namespace name (test/php-src/Zend-tests-namespace_name_reserved_keywords.test.js)
• Exceptions thrown in operand cleaning must cause leak of return value (test/php-src/Zend-tests-temporary_cleaning_013.test.js)
• Test semi-reserved words as class methods (test/php-src/Zend-tests-grammar-semi_reserved_001.test.js)
• Enum disallows static properties (test/php-src/Zend-tests-enum-no-static-properties.test.js)
• ZE2 An interface cannot have properties (test/php-src/tests-classes-interface_member.test.js)
• Bug #77530: PHP crashes when parsing '(2)::class' (test/php-src/Zend-tests-bug77530.test.js)
• Valid Unicode escape sequences: Surrogate halves (test/php-src/tests-lang-string-unicode_escape_surrogates.test.js)
• Dynamic class name in new is not supported (test/php-src/Zend-tests-constexpr-new_dynamic_class_name.test.js)
• errmsg: function cannot be declared private (test/php-src/Zend-tests-errmsg_002.test.js)
• ZE2 Late Static Binding parent::/self:: forwarding while classname doesn't (test/php-src/Zend-tests-lsb_021.test.js)
• Bug #71871: Interfaces allow final and abstract functions (test/php-src/Zend-tests-bug71871_2.test.js)
• Constant expressions with empty dimension fetch on coalesce (test/php-src/Zend-tests-constant_expressions_coalesce_empty_dim.test.js)
• ZE2 An interface method cannot be final (test/php-src/tests-classes-interface_method_final.test.js)
• ZE2 An interface method cannot be private (test/php-src/tests-classes-interface_method_private.test.js)
• using multiple access modifiers (classes) (test/php-src/Zend-tests-access_modifiers_003.test.js)
• Bug #70183 (null pointer deref (segfault) in zend_eval_const_expr) (test/php-src/Zend-tests-bug70183.test.js)
• foreach with list key (test/php-src/Zend-tests-foreach_list_003.test.js)
• ZE2 An interface method must be abstract (test/php-src/tests-classes-interface_method.test.js)
• using multiple access modifiers (abstract methods) (test/php-src/Zend-tests-access_modifiers_004.test.js)
• abstract final methods errmsg (test/php-src/Zend-tests-access_modifiers_007.test.js)
• Empty list() assignments are not allowed (test/php-src/Zend-tests-list_empty_error.test.js)
• ZE2 A final method cannot be abstract (test/php-src/tests-classes-final_abstract.test.js)
• errmsg: multiple access type modifiers are not allowed (methods) (test/php-src/Zend-tests-errmsg_010.test.js)
• Positional arguments cannot be used after argument unpacking (test/php-src/Zend-tests-arg_unpack-positional_arg_after_unpack_error.test.js)
• Cannot use static in trait insteadof method reference (test/php-src/Zend-tests-static_in_trait_insteadof_reference.test.js)
• Bug #60145 (Usage of trait's use statement inside interfaces not properly checked.) (test/php-src/Zend-tests-traits-bug60145.test.js)
• basic binary nowdoc syntax (test/php-src/Zend-tests-nowdoc_002.test.js)
• using multiple access modifiers (methods) (test/php-src/Zend-tests-access_modifiers_001.test.js)
• __HALT_COMPILER() basic test (test/php-src/Zend-tests-halt03.test.js)
• ZE2 Late Static Binding ensuring extending 'static' is not allowed (test/php-src/Zend-tests-lsb_006.test.js)
• Flexible heredoc syntax complex test 2: interpolated nested heredocs\nwith the same delimiter name (test/php-src/Zend-tests-flexible-heredoc-complex-test2.test.js)
• Flexible heredoc syntax complex test 4: interpolated variable with\nthe same delimiter name as the heredoc (test/php-src/Zend-tests-flexible-heredoc-complex-test4.test.js)
• basic binary heredoc syntax (test/php-src/Zend-tests-heredoc_002.test.js)
• Bug #47516 (nowdoc can not be embed in heredoc but can be embed in double quote) (test/php-src/Zend-tests-bug47516.test.js)
• An error should be generated when using ::class on a constant evaluated expression (test/php-src/Zend-tests-class_on_constant_evaluated_expression.test.js)
• Backed enums type can't be union (test/php-src/Zend-tests-enum-backed-type-no-union.test.js)
• errmsg: multiple access type modifiers are not allowed (properties) (test/php-src/Zend-tests-errmsg_009.test.js)
• Can use readonly as a function name (test/php-src/Zend-tests-readonly_function.test.js)
• Enum keyword can be followed by arbitrary whitespaces (test/php-src/Zend-tests-enum-keyword-whitespace.test.js)
• Class constants can be used as a class name (test/php-src/Zend-tests-varSyntax-class_constant_static_deref.test.js)
• Ensure a interface can not have private constants (test/php-src/tests-classes-interface_constant_inheritance_007.test.js)
• Bug #70912 (Null ptr dereference when class property is initialised to a dereferenced value) (test/php-src/Zend-tests-bug70912.test.js)
• Match expression with trailing comma in condition list (test/php-src/Zend-tests-match-041.test.js)
• Foo(...) in attribute in assert (test/php-src/Zend-tests-first_class_callable_assert2.test.js)
• foreach with list syntax (test/php-src/Zend-tests-foreach_list_001.test.js)
• Test semi-reserved words as static class methods (test/php-src/Zend-tests-grammar-semi_reserved_002.test.js)
• using multiple access modifiers (static methods) (test/php-src/Zend-tests-access_modifiers_006.test.js)
• Enum disallows value property (test/php-src/Zend-tests-enum-no-value-property.test.js)
• ::class on an expression cannot be used inside constant expressions (test/php-src/Zend-tests-class_on_expression_in_constant_expression.test.js)
• Cannot use multiple readonly modifiers (test/php-src/Zend-tests-readonly_props-multiple_modifiers.test.js)
• Using ::class on an object (test/php-src/Zend-tests-class_on_object.test.js)
• using multiple access modifiers (final methods) (test/php-src/Zend-tests-access_modifiers_005.test.js)
• Test to ensure semi reserved words allow deference (test/php-src/Zend-tests-grammar-regression_007.test.js)
• Enum keyword can still be used in classes, namespaces, functions and constants (test/php-src/Zend-tests-enum-keyword-no-bc-break.test.js)
• Named args after unpacking (supported) (test/php-src/Zend-tests-named_params-unpack_and_named_1.test.js)
• Invalid constant class name in nested class constant access (test/php-src/Zend-tests-invalid_const_class_name.test.js)
• Prevent abstract and final in the same class declaration (test/php-src/Zend-tests-access_modifiers_013.test.js)
• Flexible heredoc syntax complex test 3: interpolated nested heredocs\nwith the same delimiter name with different levels of indentation (test/php-src/Zend-tests-flexible-heredoc-complex-test3.test.js)
• using multiple access modifiers (attributes) (test/php-src/Zend-tests-access_modifiers_002.test.js)
• GH-7958 (Nested CallbackFilterIterator is leaking memory) (test/php-src/Zend-tests-gh7958.test.js)
• test for mbstring script_encoding for flex unsafe encoding (Shift_JIS) (test/php-src/Zend-tests-multibyte-multibyte_encoding_004.test.js)
• ZE2 A final method cannot be abstract (test/php-src/tests-classes-abstract_final.test.js)
• Flexible heredoc syntax complex test 1: interpolated nested heredocs\nwith different delimiter names (test/php-src/Zend-tests-flexible-heredoc-complex-test1.test.js)
• encoding conversion from script encoding into internal encoding (test/php-src/Zend-tests-multibyte-multibyte_encoding_005.test.js)
• Bug #76439: Don't always strip leading whitespace from heredoc T_ENCAPSED_AND_WHITESPACE tokens (error case) (test/php-src/Zend-tests-bug76439_2.test.js)
• Cannot use unpacking in attribute argument list (test/php-src/Zend-tests-attributes-026_unpack_in_args.test.js)
• First Class Callable Attribute Error (test/php-src/Zend-tests-first_class_callable_011.test.js)
• Group use declaration list should not contain leading separator (test/php-src/Zend-tests-ns_096.test.js)
• Interface cannot extend static, as it is reserved (test/php-src/Zend-tests-interface_extends_static.test.js)
• Invalid use: trailing underscore (test/php-src/Zend-tests-numeric_literal_separator_002.test.js)
• Bug #21669 ("$obj = new $this->var;" doesn't work) (test/php-src/tests-lang-bug21669.test.js)
• ZE2 Late Static Binding ensuring implementing 'static' is not allowed (test/php-src/Zend-tests-lsb_007.test.js)
• 081: bracketed namespace with nested unbracketed namespace (test/php-src/Zend-tests-ns_081.test.js)
• enum_exists (test/php-src/Zend-tests-enum-enum_exists.test.js)
• Bug #71897 (ASCII 0x7F Delete control character permitted in identifiers) (test/php-src/tests-lang-bug71897.test.js)
• Cannot catch "static" (test/php-src/Zend-tests-catch_static.test.js)
• Invalid consecutive numeric separators after binary literal (test/php-src/Zend-tests-bug78454_2.test.js)
• Flexible heredoc syntax error 9: unindented variable interpolation (test/php-src/Zend-tests-flexible-heredoc-error9.test.js)
• Zend Multibyte and ShiftJIS (test/php-src/Zend-tests-multibyte-multibyte_encoding_001.test.js)
• Class cannot use static as a trait, as it is reserved (test/php-src/Zend-tests-class_uses_static.test.js)
• Indirect modification of isref by-value return value not possible (test/php-src/Zend-tests-modify_isref_value_return.test.js)
• Alternative offset syntax should emit E_COMPILE_ERROR in const expression (test/php-src/Zend-tests-alternative_offset_syntax_compile_error_in_const_expr.test.js)
• Invalid Unicode escape sequence: Negative sign (test/php-src/tests-lang-string-unicode_escape_sign2.test.js)
• <?= cannot be used as an identifier (test/php-src/Zend-tests-short_echo_as_identifier.test.js)
• Flexible nowdoc syntax error 8: no ending token with explicit trailing space (test/php-src/Zend-tests-flexible-nowdoc-error8.test.js)
• Writing to a temporary expression is not allowed (test/php-src/Zend-tests-varSyntax-writeToTempExpr.test.js)
• Invalid Unicode escape sequence: Positive sign (test/php-src/tests-lang-string-unicode_escape_sign.test.js)
• Cannot use "namespace" as namespace name, due to conflict with ns-relative names (test/php-src/Zend-tests-namespace_name_namespace.test.js)
• eval() test (test/php-src/tests-lang-019.test.js)
• Flexible heredoc syntax error 11: show erroneous line in error message (variable interpolation) (test/php-src/Zend-tests-flexible-heredoc-error11.test.js)
• Static type is not allowed in properties (test/php-src/Zend-tests-type_declarations-static_type_property.test.js)
• Intersection type cannot be nullable (test/php-src/Zend-tests-type_declarations-intersection_types-invalid_types-invalid_nullable_type.test.js)
• Invalid Unicode escape sequence: Whitespace (test/php-src/tests-lang-string-unicode_escape_whitespace.test.js)
• Invalid Unicode escape sequence: Empty (test/php-src/tests-lang-string-unicode_escape_empty.test.js)
• Cannot use "namespace\xyz" as namespace name, due to conflict with ns-relative names (test/php-src/Zend-tests-namespace_name_namespace_start.test.js)
• Invalid Unicode escape sequence: Incomplete (test/php-src/tests-lang-string-unicode_escape_incomplete.test.js)
• Method cannot be readonly in trait alias (test/php-src/Zend-tests-readonly_props-readonly_method_trait.test.js)
• Invalid consecutive numeric separators after hex literal (test/php-src/Zend-tests-bug78454_1.test.js)
• Bug #78441 (Parse error due to heredoc identifier followed by digit) (test/php-src/Zend-tests-grammar-bug78441.test.js)
• Test typed properties type must precede first declaration in group (test/php-src/Zend-tests-type_declarations-typed_properties_025.test.js)
• Basic class support - attempting to create a reference to a class constant (test/php-src/tests-classes-constants_error_007.test.js)
• Invalid octal (test/php-src/tests-lang-invalid_octal.test.js)
• Flexible heredoc syntax error 10: unindented variable interpolation (as first value) (test/php-src/Zend-tests-flexible-heredoc-error10.test.js)

Note: it's possible that some of these are beyond the scope of this project, or are cases where the test generator is incorrectly set to expect the test to fail, or doesn't mark a test as one that should fail where it should.

[czosel]

Hi @MaartenStaa
Wow, this sounds really exciting! I’ve been wanting to reduce the amount of boilerplate in our testing setup since a while, similar to the .phpt files from php-src - but this takes it two steps further. Unfortunately I probably won’t find time to take a closer look at this until somewhen next week. For now I just wanted to share my excitement and encourage you to keep pushing this forward 💯

[czosel]

@MaartenStaa In the prettier PHP plugin, we're parsing plain php files for snapshot testing using jest setupFiles, thereby avoiding the code generation step: https://github.com/prettier/plugin-php/blob/main/tests_config/run_spec.js
Would something like that work here as well? It would be nice if we could just use the .phpt files directly without the need to generate intermediate testing code.

[cseufert]

@MaartenStaa this is really cool, should we just create a test blacklist for failing tests, so we can work towards merging this, and them working on fixing all the cases where it fails?