Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(coverage): Enable coverage on any test change (rebased with uv) #790

Merged
merged 10 commits into from
Oct 25, 2024

Conversation

winsvega
Copy link
Collaborator

@winsvega winsvega commented Sep 6, 2024

🗒️ Description

Enable coverage script on any test .py files changes:

if no new lines in converted-ethereum-tests.txt, then seek for .py files with tests and compare the coverage
from before and after PR, otherwise it compares the coverage from test repo and .py produced tests as before.

Script is reworked. use uv to run fill correctly
remove retesteth to simplify the script
use changed files github macro to detect changed files

🔗 Related Issues

Demo: #841
Rebase of: #707

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@winsvega winsvega added the type:feat type: Feature label Sep 6, 2024
@winsvega winsvega changed the title Pr coverage rebased feat(coverage): Enable coverage on any test change (rebased with uv) Sep 6, 2024
@winsvega winsvega force-pushed the pr_coverage_rebased branch 2 times, most recently from 5899822 to 90659f2 Compare September 6, 2024 09:47
@winsvega winsvega requested a review from danceratopz September 6, 2024 09:58
@winsvega winsvega force-pushed the pr_coverage_rebased branch 2 times, most recently from 38fbbd5 to eaee649 Compare September 6, 2024 10:13
@winsvega winsvega force-pushed the pr_coverage_rebased branch from eaee649 to b4edd64 Compare September 7, 2024 14:41
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one comment, I think it should simplify the logic and make it a bit less fragile.

.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @winsvega, I'd love to see this on every PR. Have a look at #839, which I think reduces the complexity on our side.

.github/workflows/coverage.yaml Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
.github/workflows/coverage.yaml Show resolved Hide resolved
@winsvega
Copy link
Collaborator Author

winsvega commented Sep 30, 2024

Merged with #839

@winsvega winsvega closed this Sep 30, 2024
@winsvega winsvega reopened this Oct 7, 2024
@winsvega winsvega force-pushed the pr_coverage_rebased branch from 95cd543 to 1767e96 Compare October 7, 2024 08:40
@winsvega
Copy link
Collaborator Author

need to exclude prague folder from coverage. but also need to check how change files behave if changes are done to prague and outside of prague.

@winsvega winsvega force-pushed the pr_coverage_rebased branch 2 times, most recently from d922729 to 3105192 Compare October 17, 2024 20:56
Copy link
Member

@danceratopz danceratopz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried out the flow locally and it's working well with the negative path matches in the Changed Files plugin 👍

A few minor comments below:

  • Is there any reason why you're not filling with xdist enabled? The flow took 21m 32s locally and the main fill call was 13 mins of this.
  • We should also ignore Osaka for now?

.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
.github/workflows/coverage.yaml Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
do not skip the coverage script for safety
@winsvega winsvega force-pushed the pr_coverage_rebased branch 3 times, most recently from e148996 to 461facc Compare October 21, 2024 15:39
@winsvega
Copy link
Collaborator Author

-n auto might be breaking >> log capture.
the script is not designed to check huge test amount.

@winsvega winsvega force-pushed the pr_coverage_rebased branch 8 times, most recently from e4aaedd to 84a1388 Compare October 22, 2024 15:47
fix uv run fill on empty file set
@winsvega winsvega force-pushed the pr_coverage_rebased branch from 84a1388 to b20eb69 Compare October 22, 2024 15:51
@winsvega winsvega requested a review from danceratopz October 22, 2024 15:57
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thanks for implementing this!

I had to create a dummy PR to verify this because act is acting funny when running this workflow for some reason. I think we can merge as is because at the end of the day we need the workflow to work on the CI and not necessarily locally, but we can eventually revisit and see if we can fix it somehow.

@marioevz marioevz merged commit f89d09b into main Oct 25, 2024
5 checks passed
@marioevz marioevz deleted the pr_coverage_rebased branch October 25, 2024 22:19
@winsvega
Copy link
Collaborator Author

Yes. Changed files gives me crypto not found error. The author says he doesn't use this in any way.

Dan doesn't have this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:feat type: Feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants