Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test timeout issues #683

Merged
merged 6 commits into from
Apr 21, 2024
Merged

Test timeout issues #683

merged 6 commits into from
Apr 21, 2024

Conversation

mk-mxp
Copy link
Contributor

@mk-mxp mk-mxp commented Apr 14, 2024

PHPUnit 10.5 assertArrayNotHasKey()

PHPUnit ^10.3 has an issue with assertArrayNotHasKey(). Only this (and the opposite assertArrayHasKey()) are affected. The issue is not listed in PHPUnit's issue list / mentioned in Pull Requests, but the issue is gone in 11.0.0.

This contains a workaround for this and enables upgrading to PHPUnit 10.5. I replaced that assertion with a functionally equivalent one, but that prints some misleading output in addition to the helpful message. So it should be reverted when upgrading to PHPUnit 11.

CI limit for test runtime

I also added a time limit of 18 (lowest value possible on my machine with equal performance to Online Test Runner) x 3 (to compensate lower CI performance) = 54 seconds for each exercise test class using timeout command. This should prevent running into performance degradation issues in the future.

As discussed in the forum there is no recommendation or guiding data for such a limit. Only guesses about "test runner is expected to be slower than GitHub CI". But the production limit of 20 seconds exists and is enforced. So I took a guess based on my measurements.

Part of #652

@mk-mxp mk-mxp requested a review from homersimpsons April 14, 2024 12:52
Copy link

This PR touches files which potentially affect the outcome of the tests of an exercise. This will cause all students' solutions to affected exercises to be re-tested.

If this PR does not affect the result of the test (or, for example, adds an edge case that is not worth rerunning all tests for), please add the following to the merge-commit message which will stops student's tests from re-running. Please copy-paste to avoid typos.

[no important files changed]

For more information, refer to the documentation. If you are unsure whether to add the message or not, please ping @exercism/maintainers-admin in a comment. Thank you!

@mk-mxp mk-mxp self-assigned this Apr 14, 2024
@mk-mxp mk-mxp added x:action/fix Fix an issue x:knowledge/intermediate Quite a bit of Exercism knowledge required x:type/ci Work on Continuous Integration (e.g. GitHub Actions workflows) x:type/content Work on content (e.g. exercises, concepts) x:size/medium Medium amount of work x:rep/medium Medium amount of reputation labels Apr 14, 2024
@mk-mxp mk-mxp removed the request for review from homersimpsons April 14, 2024 14:24
@mk-mxp
Copy link
Contributor Author

mk-mxp commented Apr 14, 2024

@homersimpsons Sorry, I thought I had figured that out right. But I somehow haven't remembered things correctly...

@mk-mxp mk-mxp closed this Apr 14, 2024
@mk-mxp mk-mxp reopened this Apr 15, 2024
@mk-mxp
Copy link
Contributor Author

mk-mxp commented Apr 15, 2024

@homersimpsons Another attempt using measured data now. I put the data into the forum, and a longish comment into the test script.

@mk-mxp mk-mxp requested a review from homersimpsons April 15, 2024 07:02
bin/test.sh Outdated Show resolved Hide resolved
Copy link
Contributor

@homersimpsons homersimpsons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your investigation. I'm proposing a new wording for the comment about the timeout. But I still approve the pull request as-is.

bin/test.sh Outdated Show resolved Hide resolved
Co-authored-by: homersimpsons <[email protected]>
@mk-mxp mk-mxp merged commit cd8a756 into exercism:main Apr 21, 2024
10 checks passed
@mk-mxp mk-mxp deleted the fix-robot-name branch April 21, 2024 13:14
tomasnorre pushed a commit to tomasnorre/exercism-php that referenced this pull request May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
x:action/fix Fix an issue x:knowledge/intermediate Quite a bit of Exercism knowledge required x:rep/medium Medium amount of reputation x:size/medium Medium amount of work x:type/ci Work on Continuous Integration (e.g. GitHub Actions workflows) x:type/content Work on content (e.g. exercises, concepts)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants