Add details about skipping tests in VCR, and when this is needed #12407

SarahFrench · 2024-11-22T19:45:31Z

Closes hashicorp/terraform-provider-google#20059

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

docs/content/develop/test/test.md

modular-magician · 2024-11-22T19:53:26Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

github-actions · 2024-11-27T09:42:37Z

@melinath This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

melinath · 2024-11-27T17:55:28Z

docs/content/develop/test/test.md

@@ -272,6 +272,51 @@ func TestSignatureAlgorithmDiffSuppress(t *testing.T) {
 }
 ```

+## Skipping tests in VCR mode


style: task-based headings should use imperative

Suggested change

## Skipping tests in VCR mode

## Skip tests in VCR replaying mode

melinath · 2024-11-27T18:01:42Z

docs/content/develop/test/test.md

@@ -272,6 +272,51 @@ func TestSignatureAlgorithmDiffSuppress(t *testing.T) {
 }
 ```

+## Skipping tests in VCR mode
+
+As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.


Suggested change

As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.

Acceptance tests are run in VCR replaying mode on PRs (using pre-recorded HTTP requests and responses) to reduce the time it takes to present results to contributors. However, not all resources or tests are possible to run in replaying mode. Incompatible tests should be skipped during VCR replaying mode. They will still run in our nightly test suite.

melinath · 2024-11-27T18:02:12Z

docs/content/develop/test/test.md

+
+As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.
+
+### When should tests be skipped in VCR mode?


style: conceptual headings should use noun phrases

Suggested change

### When should tests be skipped in VCR mode?

### Reasons that tests are skipped in VCR replaying mode

melinath · 2024-11-27T18:02:25Z

docs/content/develop/test/test.md

+
+### When should tests be skipped in VCR mode?
+
+| Problem        | Possible symptoms you'll see    | How to fix/Other info  | Skip in VCR? |


melinath · 2024-11-27T18:03:07Z

docs/content/develop/test/test.md

+| Problem        | Possible symptoms you'll see    | How to fix/Other info  | Skip in VCR? |
+| -------------- | ------------------------------- | ---------------------- |------------- |
+| Incorrect or insufficient data is present in VCR recordings to replay tests | Tests will fail with `Requested interaction not found` errors during REPLAYING mode | Make sure that you're not introducing randomness into the test, e.g. by unnecessarily using the random provider to set a resource's name.| If you cannot avoid this issue you should skip the test, but try to ensure that it cannot be fixed first.|
+| Bigtable acceptance tests aren't working in VCR mode |  `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR ofr Bigtable. |


Suggested change

| Bigtable acceptance tests aren't working in VCR mode | `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR ofr Bigtable. |

| Bigtable acceptance tests aren't working in VCR mode | `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR for Bigtable. |

melinath · 2024-11-27T18:03:53Z

docs/content/develop/test/test.md

+| -------------- | ------------------------------- | ---------------------- |------------- |
+| Incorrect or insufficient data is present in VCR recordings to replay tests | Tests will fail with `Requested interaction not found` errors during REPLAYING mode | Make sure that you're not introducing randomness into the test, e.g. by unnecessarily using the random provider to set a resource's name.| If you cannot avoid this issue you should skip the test, but try to ensure that it cannot be fixed first.|
+| Bigtable acceptance tests aren't working in VCR mode |  `Requested interaction not found` errors during REPLAYING mode | Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.| Skip the test in VCR ofr Bigtable. |
+| Using multiple provider aliases doesn't work in VCR | You may have two instances of the google provider in the test config but one of them doesn't seem to be using its provider arguments, e.g. using the wrong default project. | See this GitHub issue: https://github.com/hashicorp/terraform-provider-google/issues/20019 . The problem is that, due to how the VCR system works, one provider instance will be configured and the other will be forced to reuse the first instance's configuration, despite them being given different provider arguments. |  Skip the test in VCR is using aliases is unavoidable. |


style: avoid e.g.

Suggested change

| Using multiple provider aliases doesn't work in VCR | You may have two instances of the google provider in the test config but one of them doesn't seem to be using its provider arguments, e.g. using the wrong default project. | See this GitHub issue: https://github.com/hashicorp/terraform-provider-google/issues/20019 . The problem is that, due to how the VCR system works, one provider instance will be configured and the other will be forced to reuse the first instance's configuration, despite them being given different provider arguments. | Skip the test in VCR is using aliases is unavoidable. |

| Using multiple provider aliases doesn't work in VCR | You may have two instances of the google provider in the test config but one of them doesn't seem to be using its provider arguments - for example, using the wrong default project. | See this GitHub issue: https://github.com/hashicorp/terraform-provider-google/issues/20019 . The problem is that, due to how the VCR system works, one provider instance will be configured and the other will be forced to reuse the first instance's configuration, despite them being given different provider arguments. | Skip the test in VCR is using aliases is unavoidable. |

melinath · 2024-11-27T18:06:41Z

docs/content/develop/test/test.md

+- When the provider is configured it will use credentials to obtain access tokens from GCP
+- Some acceptance tests use bootstrapping functions that ensure long-lived resources are present in a testing project before the provider is tested.
+
+Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.


Suggested change

Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.

These tests can still run in VCR replaying mode; however, REPLAYING mode can't be used as a way to completely avoid HTTP traffic generally or with GCP APIs.

melinath · 2024-11-27T18:07:42Z

docs/content/develop/test/test.md

+
+Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.
+
+### How can tests be skipped in VCR mode?


style: prefer imperative headings

Suggested change

### How can tests be skipped in VCR mode?

### Skip tests in VCR replaying mode

Since this duplicates the parent heading, I'd be inclined to merge this content into the parent heading (that is, above the "reasons" section)

melinath · 2024-11-27T18:13:02Z

docs/content/develop/test/test.md

+
+### How can tests be skipped in VCR mode?
+
+You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue.


Suggested change

You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue.

If you skip a test in VCR mode, include a code comment explaining the reason for skipping (for example, a link to a GitHub issue.)

melinath · 2024-11-27T18:14:09Z

docs/content/develop/test/test.md

+Skipping acceptance tests that are generated from example files can be achieved by adding `skip_vcr: true` in the example's YAML:
+
+```yaml
+examples:
+  - name: 'bigtable_app_profile_anycluster'
+    ...
+
+    # bigtable instance does not use the shared HTTP client, this test creates an instance
+    skip_vcr: true
+```
+
+Skipping acceptance tests that are handwritten can be achieved by adding `acctest.SkipIfVcr(t)` at the start of the test:
+
+```go
+func TestAccPubsubTopic_update(t *testing.T) {
+      acctest.SkipIfVcr(t)
+      acctest.VcrTest(t, resource.TestCase{ ... }
+}
+```


This would be great to wrap in tabs like

magic-modules/docs/content/develop/test/test.md

Line 46 in 1a21cf3

{{< tabs "create" >}}

SarahFrench · 2024-11-27T23:24:30Z

Thanks for the review, and the link to the developer docs style guide! I'll work through your comments tomorrow and this can be wrapped up after Thanksgiving 😁

Add details about skipping tests in VCR, and when this is needed

50f2342

SarahFrench commented Nov 22, 2024

View reviewed changes

docs/content/develop/test/test.md Outdated Show resolved Hide resolved

Update docs/content/develop/test/test.md

1a21cf3

SarahFrench requested a review from melinath November 22, 2024 19:46

SarahFrench marked this pull request as ready for review November 22, 2024 19:47

SarahFrench mentioned this pull request Nov 22, 2024

Update link about VCR tests on automated PR comments #12405

Merged

This comment was marked as duplicate.

Sign in to view

melinath requested changes Nov 27, 2024

View reviewed changes

melinath requested review from nityaravi and removed request for nityaravi November 27, 2024 18:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add details about skipping tests in VCR, and when this is needed #12407

Add details about skipping tests in VCR, and when this is needed #12407

SarahFrench commented Nov 22, 2024 •

edited

Loading

modular-magician commented Nov 22, 2024

This comment was marked as duplicate.

github-actions bot commented Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

melinath Nov 27, 2024

SarahFrench commented Nov 27, 2024

	## Skipping tests in VCR mode
	## Skip tests in VCR replaying mode

	As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.
	Acceptance tests are run in VCR replaying mode on PRs (using pre-recorded HTTP requests and responses) to reduce the time it takes to present results to contributors. However, not all resources or tests are possible to run in replaying mode. Incompatible tests should be skipped during VCR replaying mode. They will still run in our nightly test suite.


		As described above, acceptance tests can be run in VCR modes where HTTP requests are either recorded or are replayed. This system can fail in some scenarios, so we will oftern flag tests that are incompatible with VCR so that they're skipped. This means the tests will not run automatically on PRs (in VCR mode) but the tests will still run in our nightly test suite.

		### When should tests be skipped in VCR mode?

	### When should tests be skipped in VCR mode?
	### Reasons that tests are skipped in VCR replaying mode


		### When should tests be skipped in VCR mode?

		\| Problem \| Possible symptoms you'll see \| How to fix/Other info \| Skip in VCR? \|

	\| Bigtable acceptance tests aren't working in VCR mode \| `Requested interaction not found` errors during REPLAYING mode \| Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.\| Skip the test in VCR ofr Bigtable. \|
	\| Bigtable acceptance tests aren't working in VCR mode \| `Requested interaction not found` errors during REPLAYING mode \| Currently the provider uses a separate client than the rest of the provider to interact with the Bigtable API. As HTTP traffic to the Bigtable API doesn't go via the shared client it cannot be recorded in RECORDING mode.\| Skip the test in VCR for Bigtable. \|

	Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.
	These tests can still run in VCR replaying mode; however, REPLAYING mode can't be used as a way to completely avoid HTTP traffic generally or with GCP APIs.


		Due to this you cannot use REPLAYING mode as a way to completely avoid HTTP traffic generally or with GCP APIs.

		### How can tests be skipped in VCR mode?

	### How can tests be skipped in VCR mode?
	### Skip tests in VCR replaying mode


		### How can tests be skipped in VCR mode?

		You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue.

	You can make generated and handwritten tests be skipped in VCR mode. If possible, please include a code comment explaining the reason for skipping, or link to a GitHub issue.
	If you skip a test in VCR mode, include a code comment explaining the reason for skipping (for example, a link to a GitHub issue.)

Add details about skipping tests in VCR, and when this is needed #12407

Are you sure you want to change the base?

Add details about skipping tests in VCR, and when this is needed #12407

Conversation

SarahFrench commented Nov 22, 2024 • edited Loading

modular-magician commented Nov 22, 2024

Diff report

This comment was marked as duplicate.

github-actions bot commented Nov 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SarahFrench commented Nov 27, 2024

SarahFrench commented Nov 22, 2024 •

edited

Loading