Add QIT Custom E2E support and migrate tests

[iamdharmesh]

Fixes #3530

Changes proposed in this Pull Request:

This PR introduces initial support for QIT Custom E2E tests and migrates existing E2E tests to the QIT framework, enabling them to run in the QIT environment. As part of this migration, the old test environment has been removed, as QIT provides its own environment for running tests.

This is the first step towards adding integration tests to the workflow. QIT Custom E2E tests allow us to run tests for other plugins. This enables us to check plugin compatibility by running tests with specific plugins activated and configured or testing plugins while other plugins are active.

Key Changes

• Support for QIT Custom E2E Tests: Added QIT integration to enable Custom E2E testing.
• Migration of Existing E2E Tests: Migrated current tests to QIT, including:
  • Removal of the custom Docker-based test environment setup.
  • Updates to GitHub Action workflow.
• Initial Compatibility Testing: Implemented compatibility tests with the "WooCommerce PayPal Payments" plugin in GitHub Action.
• New Workflow for Upload test tags: Added a GitHub Action workflow to upload test tags to QIT upon release. Tests are categorized into:
  • default: Standard E2E tests (excludes legacy experience tests).
  • legacy: Tests for legacy experience.
  • subscriptions-tests: Tests for subscriptions functionality.

Potential Improvements for the Next Iteration

• Improved Bootstrap Process: The current QIT CLI uses bootstrap.sh for plugin setup. The QIT team is working on improving this to support running tests as part of the bootstrap process. Once available, we can enhance our bootstrap workflow accordingly.
• Better Developer Experience for Writing Tests: Currently, updating test files requires restarting the test runner. Exploring a solution to apply changes without restarting the runner will improve developer efficiency.

Questions/Things to Discuss:

1. I’ve updated the E2E tests GitHub Action to run compatibility tests with the "WooCommerce PayPal Payments" plugin on every PR. Should this be changed to run less frequently (e.g., on release) and limit PR checks to only the E2E tests for this plugin?

Testing instructions

1. Run npm install && composer install && npm run build.
2. Copy the file /tests/e2e/config/local.env.example to /tests/e2e/config/local.env.
3. Add the stripe credentials on the local.env file.
4. Run npm run test:e2e-local and make sure that tests are running properly and are passing.
5. Also, Make sure that the "E2E tests" actions are passing on the PR.

---

• Covered with tests (or have a good reason not to test in description ☝️)
• Added changelog entry in both changelog.txt and readme.txt (or does not apply)
• Tested on mobile (or does not apply)

Post merge

• Added testing instructions to the Release Testing Instructions wiki page (or does not apply)
• Added the needs docs label (or does not apply)
• Included this PR in the Release Thread scope (or does not apply)

[vikrampm1]

@wjrosa this PR is ready for your review!

[wjrosa]

@wjrosa this PR is ready for your review!

Switching myself with @annemirasol as our porter this week.

[Luc45]

Thanks for pinging for review, I'm taking a look at it now and it's looking great overall.

Minor nit, but just to keep things dry you should be able to remove npx playwright install from the postinstall command in package.json, now that this is handled by QIT.

[Luc45]

This worked great for me. I've just cloned this repo, added my Stripe keys on the env file, and ran these:

• nvm use
• npm i
• npm run build
• npm run test:local

And this is the test report that it generated.

I've just noticed a little bit of flakiness in some of the assertions that can be improved in the future, just for increased reliability in the tests themselves.

QIT improved orchestration is in code review and will be merged sometime this week, when it does, we'd just need to rename bootstrap.sh to setup.sh and entrypoint.js to setup.js.

[annemirasol]

@iamdharmesh @vikrampm1 To avoid impacting development that are currently underway for Stripe and Paypal, we created a new repo for hosting the new QIT tests. Could we move these tests to https://github.com/woocommerce/woocommerce-gateway-integration-qit, please?

cc @diegocurbelo