-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add details about skipping tests in VCR, and when this is needed #12407
base: main
Are you sure you want to change the base?
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -272,6 +272,51 @@ func TestSignatureAlgorithmDiffSuppress(t *testing.T) { | |||||
} | ||||||
``` | ||||||
|
||||||
## Skipping tests in VCR mode | ||||||
|
||||||
As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
### When should tests be skipped in VCR mode? | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. style: conceptual headings should use noun phrases
Suggested change
|
||||||
|
||||||
| Problem | Possible symptoms you'll see | How to fix/Other info | Skip in VCR? | | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| -------------- | ------------------------------- | ---------------------- |------------- | | ||||||
| Incorrect or insufficient data is present in VCR recordings to replay tests | Tests will fail with `Requested interaction not found` errors during REPLAYING mode | Make sure that you're not introducing randomness into the test, e.g. by unnecessarily using the random provider to set a resource's name.| If you cannot avoid this issue you should skip the test, but try to ensure that it cannot be fixed first.| | ||||||
| Bigtable acceptance tests aren't working in VCR mode | `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR ofr Bigtable. | | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| Using multiple provider aliases doesn't work in VCR | You may have two instances of the google provider in the test config but one of them doesn't seem to be using its provider arguments, e.g. using the wrong default project. | See this GitHub issue: https://github.com/hashicorp/terraform-provider-google/issues/20019 . The problem is that, due to how the VCR system works, one provider instance will be configured and the other will be forced to reuse the other instance's configuration, despite them being given different provider arguments. | Skip the test in VCR is using aliases is unavoidable. | | ||||||
SarahFrench marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| Using multiple versions of the google/google-beta provider in a single test isn't working in VCR | Unexpected test failures during REPLAYING mode in tests with `ExternalProviders` pulling in past versions of the google/google-beta provider | When ExternalProviders is used to pulling in other versions of the provider, any HTTP traffic through the external provider will not be recorded. If the HTTP traffic produces an unexpected result or returns an API error then the test will fail in REPLAYING mode. | Skip the test in VCR when testing the current provider behaviour versus previous released versions. | | ||||||
|
||||||
Some additional things to bear in mind are that VCR tests in REPLAYING mode will still interact with GCP APIs somewhat. For example: | ||||||
|
||||||
- When the provider is configured it will use credentials to obtain access tokens from GCP | ||||||
- Some acceptance tests use bootstrapping functions that ensure long-lived resources are present in a testing project before the provider is tested. | ||||||
|
||||||
Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
### How can tests be skipped in VCR mode? | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. style: prefer imperative headings
Suggested change
Since this duplicates the parent heading, I'd be inclined to merge this content into the parent heading (that is, above the "reasons" section) |
||||||
|
||||||
You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
Skipping acceptance tests that are generated from example files can be achieved by adding `skip_vcr: true` in the example's YAML: | ||||||
|
||||||
```yaml | ||||||
examples: | ||||||
- name: 'bigtable_app_profile_anycluster' | ||||||
... | ||||||
|
||||||
# bigtable instance does not use the shared HTTP client, this test creates an instance | ||||||
skip_vcr: true | ||||||
``` | ||||||
|
||||||
Skipping acceptance tests that are handwritten can be achieved by adding `acctest.SkipIfVcr(t)` at the start of the test: | ||||||
|
||||||
```go | ||||||
func TestAccPubsubTopic_update(t *testing.T) { | ||||||
acctest.SkipIfVcr(t) | ||||||
acctest.VcrTest(t, resource.TestCase{ ... } | ||||||
} | ||||||
``` | ||||||
Comment on lines
+299
to
+317
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This would be great to wrap in tabs like
|
||||||
|
||||||
|
||||||
## What's next? | ||||||
|
||||||
- [Run your tests]({{< ref "/develop/test/run-tests" >}}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
style: task-based headings should use imperative