Percy Visual Regression Optimisation

[siggerzz]

PIE Monorepo Percy Usage by component

As you'll know, we rely on Percy heavily for the testing of PIE and Fozzie components. While the bulk of visual regression tests live in Fozzie Components, PIE is currently consuming the vast majority of our JET-wide 400K monthly allowance.

Component name	Unique Screenshots	Tested Viewports	Chrome	Firefox	Edge	Safari	Total Screenshots
pie-docs	46	414px / 768px / 1280px	✅	✅	❌	✅	414
pie-button	11	414px / 1280px	✅	✅	❌	✅	66
pie-icon-button	5	414px / 1280px	✅	✅	❌	✅	30
pie-modal	33	414px / 1280px	✅	✅	❌	✅	198
pie-divider	1	414px / 1280px	✅	✅	❌	✅	6

This means that in a worst case scenario PR that affects all components, a single commit will produce a total of 714 screenshots.

We currently take the following measures to reduce our usage across both of our repos

Run Visual Tests based on branch changes

• Use Turborepo's --filter functionality to compare the PR branch to main, and only run visual tests against changed components and their dependants.

For example, below you can see a change to pie-button also triggers visual tests for pie-modal because this component depends on button.

*The above is an example Turborepo dry run for test:visual once a change has been made to pie-button yarn test:visual --filter='!./apps/**/*' --filter=...\[origin/main\] --graph=index.html

Only run Visual Tests against non-draft PR's

Currently we check in CI whether the Pull Request is in draft, and if so, skip execution.

Optimisations we can make to further reduce usage

Better understand how changes affect what Turborepo determines as a changed package

Whilst our --filter functionality works well, Turborepo can occasionally trigger a run of visual tests of components that are seemingly unrelated to any changes in the PR.

Example of this behaviour

In PR #668, a new pie-css package was introduced to the monorepo. As expected, no visual tests were run. However, when main was merged into this branch, it triggered all visual tests to run

Only update baseline images of components and dependants that changed in the last commit.

Currently when we merge to main, visual test baseline images are updated for all components. We can probably use Turborepo --filter functionality to only update the main baseline images of components that were last commited to main

Extend non-draft functionality

A scenario that comes to mind that can trigger unnecessary runs is as follows:

• PR author makes a PR ready to review and bring it out of draft
• Reviewers request changes that result in a chain of new commits
• PR author doesn't convert the PR back to draft, resulting in visual test runs for the subsequent commits.

This could be rectified by also disabling visual regression runs against PR's that have the work in progress label. A safety check can also be added to CI to prevent the merging of PR's that have this label, meaning there's no risk of a PR getting merged without being visually tested.

[siggerzz]

Expanding on following

Only update baseline images of components and dependants that changed in the last commit

After some thought, I think we might be able to enable Turborepo's remote caching, with some tweaks.

My original worry was that by enabling remote caching, the baseline images wouldn't get updated.

By running the following command in main we should be able to force execution of visual tests that changed in the last commit:

yarn test:visual --filter=[HEAD^1] --force

[raoufswe]

Thanks, Ben, for setting this up!

I love the idea of using filters and only running visual tests on the specific changes. I think we also should consider taking just one screenshot of each component instead of separate snapshots for each component story, or be very conscious of the way we write our stories.

One more idea is to handle the visual regression tests manually. In GitLab, you can set up an optional step in your pipeline that requires manual triggering. Marking pull requests as drafts could be a good way to do it but this manual approach ensures that we won't accidentally leave the tests running and also guarantees we making each snapshot we use counts!

[jamieomaguire]

I think it's a really good idea having a manual trigger for the visual tests :)

[jamieomaguire]

Thanks @siggerzz !

You referred to that pie-css PR that triggered all the tests - this would definitely be a great one to solve as it'll only get worse as our list of components grows.

I wonder if we could find a way to assess whether or not changes to the yarn.lock file actually warrant visual tests to run.