Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add details about skipping tests in VCR, and when this is needed #12407

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/content/develop/test/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,51 @@ func TestSignatureAlgorithmDiffSuppress(t *testing.T) {
}
```

## Skipping tests in VCR mode
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: task-based headings should use imperative

Suggested change
## Skipping tests in VCR mode
## Skip tests in VCR replaying mode


As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.
Acceptance tests are run in VCR replaying mode on PRs (using pre-recorded HTTP requests and responses) to reduce the time it takes to present results to contributors. However, not all resources or tests are possible to run in replaying mode. Incompatible tests should be skipped during VCR replaying mode. They will still run in our nightly test suite.


### When should tests be skipped in VCR mode?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: conceptual headings should use noun phrases

Suggested change
### When should tests be skipped in VCR mode?
### Reasons that tests are skipped in VCR replaying mode


| Problem | Possible symptoms you'll see | How to fix/Other info | Skip in VCR? |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| Problem | Possible symptoms you'll see | How to fix/Other info | Skip in VCR? |
| Problem | Possible symptoms you'll see | How to fix/Other info | Skip in VCR replaying? |

| -------------- | ------------------------------- | ---------------------- |------------- |
| Incorrect or insufficient data is present in VCR recordings to replay tests | Tests will fail with `Requested interaction not found` errors during REPLAYING mode | Make sure that you're not introducing randomness into the test, e.g. by unnecessarily using the random provider to set a resource's name.| If you cannot avoid this issue you should skip the test, but try to ensure that it cannot be fixed first.|
| Bigtable acceptance tests aren't working in VCR mode | `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR ofr Bigtable. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| Bigtable acceptance tests aren't working in VCR mode | `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR ofr Bigtable. |
| Bigtable acceptance tests aren't working in VCR mode | `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR for Bigtable. |

| Using multiple provider aliases doesn't work in VCR | You may have two instances of the google provider in the test config but one of them doesn't seem to be using its provider arguments, e.g. using the wrong default project. | See this GitHub issue: https://github.com/hashicorp/terraform-provider-google/issues/20019 . The problem is that, due to how the VCR system works, one provider instance will be configured and the other will be forced to reuse the other instance's configuration, despite them being given different provider arguments. | Skip the test in VCR is using aliases is unavoidable. |
SarahFrench marked this conversation as resolved.
Show resolved Hide resolved
| Using multiple versions of the google/google-beta provider in a single test isn't working in VCR | Unexpected test failures during REPLAYING mode in tests with `ExternalProviders` pulling in past versions of the google/google-beta provider | When ExternalProviders is used to pulling in other versions of the provider, any HTTP traffic through the external provider will not be recorded. If the HTTP traffic produces an unexpected result or returns an API error then the test will fail in REPLAYING mode. | Skip the test in VCR when testing the current provider behaviour versus previous released versions. |

Some additional things to bear in mind are that VCR tests in REPLAYING mode will still interact with GCP APIs somewhat. For example:

- When the provider is configured it will use credentials to obtain access tokens from GCP
- Some acceptance tests use bootstrapping functions that ensure long-lived resources are present in a testing project before the provider is tested.

Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.
These tests can still run in VCR replaying mode; however, REPLAYING mode can't be used as a way to completely avoid HTTP traffic generally or with GCP APIs.


### How can tests be skipped in VCR mode?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: prefer imperative headings

Suggested change
### How can tests be skipped in VCR mode?
### Skip tests in VCR replaying mode

Since this duplicates the parent heading, I'd be inclined to merge this content into the parent heading (that is, above the "reasons" section)


You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue.
If you skip a test in VCR mode, include a code comment explaining the reason for skipping (for example, a link to a GitHub issue.)


Skipping acceptance tests that are generated from example files can be achieved by adding `skip_vcr: true` in the example's YAML:

```yaml
examples:
- name: 'bigtable_app_profile_anycluster'
...

# bigtable instance does not use the shared HTTP client, this test creates an instance
skip_vcr: true
```

Skipping acceptance tests that are handwritten can be achieved by adding `acctest.SkipIfVcr(t)` at the start of the test:

```go
func TestAccPubsubTopic_update(t *testing.T) {
acctest.SkipIfVcr(t)
acctest.VcrTest(t, resource.TestCase{ ... }
}
```
Comment on lines +299 to +317
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be great to wrap in tabs like

{{< tabs "create" >}}



## What's next?

- [Run your tests]({{< ref "/develop/test/run-tests" >}})
Loading