Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: 5.24 provider stalls during plan phase #34257

Closed
mire3212 opened this issue Nov 3, 2023 · 10 comments · Fixed by #34300
Closed

[Bug]: 5.24 provider stalls during plan phase #34257

mire3212 opened this issue Nov 3, 2023 · 10 comments · Fixed by #34300
Assignees
@gdavison
Labels
authentication Pertains to authentication; to the provider itself of otherwise. bug Addresses a defect in current functionality. prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. provider Pertains to the provider itself, rather than any interaction with AWS.
Milestone
v5.25.0

Comments

@mire3212
Copy link

mire3212 commented Nov 3, 2023 •
edited
Loading

Terraform Core Version

1.5.7

AWS Provider Version

5.24.0

Affected Resource(s)

Terraform stalls while performing terraform plan. The init and validation phases work without issue, but using plan will just sit and stall and never complete.

Rolling back to provider 5.23.0 allows the plan to complete as expected.

Expected Behavior

terraform plan should complete and report the expected actions needed.

Actual Behavior

CPU is consumed but nothing happens.

Relevant Error/Panic Output Snippet

The last few lines of the log output:

2023-11-03T14:59:58.396-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: plugin address: address=/var/folders/n4/0mw47cfx2l7_y5wg1l72w1xh0000gn/T/plugin2705565401 network=unix timestamp=2023-11-03T14:59:58.396-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Configuring Terraform AWS Provider: @caller=github.com/hashicorp/terraform-provider-aws/internal/conns/config.go:125 @module=aws tf_mux_provider=*schema.GRPCProviderServer tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 tf_rpc=ConfigureProvider tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2023-11-03T14:59:58.879-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Resolving credentials provider: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:47 @module=aws.aws-base tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ConfigureProvider timestamp=2023-11-03T14:59:58.879-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Using profile: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 tf_aws.profile.source=provider tf_mux_provider=*schema.GRPCProviderServer tf_aws.profile=<redacted> tf_rpc=ConfigureProvider @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:47 @module=aws.aws-base timestamp=2023-11-03T14:59:58.879-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Loading profile: tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ConfigureProvider @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:47 @module=aws.aws-base tf_aws.profile=<redacted> tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 timestamp=2023-11-03T14:59:58.879-0500

Terraform Configuration Files

Any configuration

Steps to Reproduce

Create a new configuration with the latest provider code and attempt to execute terraform plan

Debug Output

% TF_LOG=DEBUG terraform plan
2023-11-03T14:59:56.130-0500 [INFO]  Terraform version: 1.5.7
2023-11-03T14:59:56.130-0500 [DEBUG] using github.com/hashicorp/go-tfe v1.26.0
2023-11-03T14:59:56.130-0500 [DEBUG] using github.com/hashicorp/hcl/v2 v2.16.2
2023-11-03T14:59:56.130-0500 [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.0
2023-11-03T14:59:56.130-0500 [DEBUG] using github.com/zclconf/go-cty v1.12.2
2023-11-03T14:59:56.130-0500 [INFO]  Go runtime version: go1.21.1
2023-11-03T14:59:56.130-0500 [INFO]  CLI args: []string{"terraform", "plan"}
2023-11-03T14:59:56.130-0500 [DEBUG] Attempting to open CLI config file: /Users/<redacted>/.terraformrc
2023-11-03T14:59:56.130-0500 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-11-03T14:59:56.130-0500 [INFO]  Loading CLI configuration from /Users/<redacted>/.terraform.d/credentials.tfrc.json
2023-11-03T14:59:56.130-0500 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-11-03T14:59:56.130-0500 [DEBUG] ignoring non-existing provider search directory /Users/<redacted>/.terraform.d/plugins
2023-11-03T14:59:56.130-0500 [DEBUG] ignoring non-existing provider search directory /Users/<redacted>/Library/Application Support/io.terraform/plugins
2023-11-03T14:59:56.130-0500 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2023-11-03T14:59:56.131-0500 [INFO]  CLI command args: []string{"plan"}
2023-11-03T14:59:56.147-0500 [DEBUG] New state was assigned lineage "b82b46ae-9d50-efa4-f3cd-6ddc25464d87"
2023-11-03T14:59:56.993-0500 [DEBUG] checking for provisioner in "."
2023-11-03T14:59:56.995-0500 [DEBUG] checking for provisioner in "/usr/local/bin"
2023-11-03T14:59:56.995-0500 [INFO]  backend/local: starting Plan operation
2023-11-03T14:59:57.000-0500 [DEBUG] created provider logger: level=debug
2023-11-03T14:59:57.000-0500 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:59:57.010-0500 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5]
2023-11-03T14:59:57.017-0500 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 pid=4156
2023-11-03T14:59:57.017-0500 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5
2023-11-03T14:59:57.104-0500 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:59:57.104-0500
2023-11-03T14:59:57.120-0500 [DEBUG] provider: using plugin: version=5
2023-11-03T14:59:57.121-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: plugin address: network=unix address=/var/folders/n4/0mw47cfx2l7_y5wg1l72w1xh0000gn/T/plugin2844671496 timestamp=2023-11-03T14:59:57.120-0500
2023-11-03T14:59:57.652-0500 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-11-03T14:59:57.656-0500 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 pid=4156
2023-11-03T14:59:57.656-0500 [DEBUG] provider: plugin exited
2023-11-03T14:59:57.656-0500 [DEBUG] Building and walking validate graph
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_kms_key.aws_ebs" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_vpc.shared_prod" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.ubuntu_20" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_amazon_linux_2" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.ubuntu_22" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_iam_instance_profile.ssm_quick_setup" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.amazon_linux_2" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_security_group.allow_internal" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_centos" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.centos_7" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_key_pair.<redacted>" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_subnet.subnet_a" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_ubuntu_20" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.657-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_ubuntu_22" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.centos_7" references: [module.ec2.data.aws_iam_instance_profile.ssm_quick_setup module.ec2.data.aws_subnet.subnet_a module.ec2.aws_security_group.allow_internal module.ec2.data.aws_ami.golden_centos module.ec2.aws_key_pair.<redacted> module.ec2.data.aws_kms_key.aws_ebs]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_security_group.allow_internal" references: [module.ec2.data.aws_vpc.shared_prod]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2 (close)" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2 (expand)" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_centos" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_kms_key.aws_ebs" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_vpc.shared_prod" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_ubuntu_20" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_iam_instance_profile.ssm_quick_setup" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.local.amis (expand)" references: [module.ec2.data.aws_ami.golden_amazon_linux_2 module.ec2.data.aws_ami.golden_ubuntu_20 module.ec2.data.aws_ami.golden_ubuntu_22 module.ec2.data.aws_ami.golden_centos]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_key_pair.<redacted>" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.ubuntu_22" references: [module.ec2.data.aws_ami.golden_ubuntu_22 module.ec2.aws_key_pair.<redacted> module.ec2.data.aws_iam_instance_profile.ssm_quick_setup module.ec2.data.aws_subnet.subnet_a module.ec2.aws_security_group.allow_internal module.ec2.data.aws_kms_key.aws_ebs]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_subnet.subnet_a" references: [module.ec2.data.aws_vpc.shared_prod]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_ubuntu_22" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/aws\"]" references: []
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.ubuntu_20" references: [module.ec2.aws_security_group.allow_internal module.ec2.data.aws_ami.golden_ubuntu_20 module.ec2.aws_key_pair.<redacted> module.ec2.data.aws_iam_instance_profile.ssm_quick_setup module.ec2.data.aws_subnet.subnet_a module.ec2.data.aws_kms_key.aws_ebs]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.amazon_linux_2" references: [module.ec2.data.aws_iam_instance_profile.ssm_quick_setup module.ec2.data.aws_subnet.subnet_a module.ec2.aws_security_group.allow_internal module.ec2.data.aws_ami.golden_amazon_linux_2 module.ec2.aws_key_pair.<redacted> module.ec2.data.aws_kms_key.aws_ebs]
2023-11-03T14:59:57.658-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_amazon_linux_2" references: []
2023-11-03T14:59:57.660-0500 [DEBUG] Starting graph walk: walkValidate
2023-11-03T14:59:57.660-0500 [DEBUG] created provider logger: level=debug
2023-11-03T14:59:57.660-0500 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:59:57.664-0500 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5]
2023-11-03T14:59:57.672-0500 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 pid=4157
2023-11-03T14:59:57.672-0500 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5
2023-11-03T14:59:57.761-0500 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:59:57.761-0500
2023-11-03T14:59:57.777-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: plugin address: address=/var/folders/n4/0mw47cfx2l7_y5wg1l72w1xh0000gn/T/plugin1001438033 network=unix timestamp=2023-11-03T14:59:57.777-0500
2023-11-03T14:59:57.777-0500 [DEBUG] provider: using plugin: version=5
2023-11-03T14:59:58.274-0500 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2023-11-03T14:59:58.278-0500 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 pid=4157
2023-11-03T14:59:58.278-0500 [DEBUG] provider: plugin exited
2023-11-03T14:59:58.278-0500 [INFO]  backend/local: plan calling Plan
2023-11-03T14:59:58.278-0500 [DEBUG] Building and walking plan graph for NormalMode
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_centos (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_iam_instance_profile.ssm_quick_setup (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_key_pair.<redacted> (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_security_group.allow_internal (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_kms_key.aws_ebs (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.centos_7 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.amazon_linux_2 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.ubuntu_22 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_amazon_linux_2 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_ubuntu_20 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_subnet.subnet_a (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_ami.golden_ubuntu_22 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.aws_instance.ubuntu_20 (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ProviderTransformer: "module.ec2.data.aws_vpc.shared_prod (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/hashicorp/aws"]
2023-11-03T14:59:58.279-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_security_group.allow_internal (expand)" references: [module.ec2.data.aws_vpc.shared_prod (expand)]
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_amazon_linux_2 (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_ubuntu_20 (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_subnet.subnet_a (expand)" references: [module.ec2.data.aws_vpc.shared_prod (expand)]
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2 (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.centos_7 (expand)" references: [module.ec2.data.aws_iam_instance_profile.ssm_quick_setup (expand) module.ec2.data.aws_subnet.subnet_a (expand) module.ec2.aws_security_group.allow_internal (expand) module.ec2.data.aws_ami.golden_centos (expand) module.ec2.aws_key_pair.<redacted> (expand) module.ec2.data.aws_kms_key.aws_ebs (expand)]
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_ubuntu_22 (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.ubuntu_20 (expand)" references: [module.ec2.aws_key_pair.<redacted> (expand) module.ec2.data.aws_iam_instance_profile.ssm_quick_setup (expand) module.ec2.data.aws_subnet.subnet_a (expand) module.ec2.aws_security_group.allow_internal (expand) module.ec2.data.aws_ami.golden_ubuntu_20 (expand) module.ec2.data.aws_kms_key.aws_ebs (expand)]
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.amazon_linux_2 (expand)" references: [module.ec2.data.aws_iam_instance_profile.ssm_quick_setup (expand) module.ec2.data.aws_subnet.subnet_a (expand) module.ec2.aws_security_group.allow_internal (expand) module.ec2.data.aws_ami.golden_amazon_linux_2 (expand) module.ec2.aws_key_pair.<redacted> (expand) module.ec2.data.aws_kms_key.aws_ebs (expand)]
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_ami.golden_centos (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_iam_instance_profile.ssm_quick_setup (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/aws\"]" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2 (close)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_key_pair.<redacted> (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.aws_instance.ubuntu_22 (expand)" references: [module.ec2.data.aws_iam_instance_profile.ssm_quick_setup (expand) module.ec2.data.aws_subnet.subnet_a (expand) module.ec2.aws_security_group.allow_internal (expand) module.ec2.data.aws_ami.golden_ubuntu_22 (expand) module.ec2.aws_key_pair.<redacted> (expand) module.ec2.data.aws_kms_key.aws_ebs (expand)]
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_kms_key.aws_ebs (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.data.aws_vpc.shared_prod (expand)" references: []
2023-11-03T14:59:58.280-0500 [DEBUG] ReferenceTransformer: "module.ec2.local.amis (expand)" references: [module.ec2.data.aws_ami.golden_amazon_linux_2 (expand) module.ec2.data.aws_ami.golden_ubuntu_20 (expand) module.ec2.data.aws_ami.golden_ubuntu_22 (expand) module.ec2.data.aws_ami.golden_centos (expand)]
2023-11-03T14:59:58.282-0500 [DEBUG] Starting graph walk: walkPlan
2023-11-03T14:59:58.282-0500 [DEBUG] created provider logger: level=debug
2023-11-03T14:59:58.282-0500 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:59:58.286-0500 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5]
2023-11-03T14:59:58.294-0500 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5 pid=4158
2023-11-03T14:59:58.294-0500 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.24.0/darwin_amd64/terraform-provider-aws_v5.24.0_x5
2023-11-03T14:59:58.381-0500 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:59:58.381-0500
2023-11-03T14:59:58.396-0500 [DEBUG] provider: using plugin: version=5
2023-11-03T14:59:58.396-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: plugin address: address=/var/folders/n4/0mw47cfx2l7_y5wg1l72w1xh0000gn/T/plugin2705565401 network=unix timestamp=2023-11-03T14:59:58.396-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Configuring Terraform AWS Provider: @caller=github.com/hashicorp/terraform-provider-aws/internal/conns/config.go:125 @module=aws tf_mux_provider=*schema.GRPCProviderServer tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 tf_rpc=ConfigureProvider tf_provider_addr=registry.terraform.io/hashicorp/aws timestamp=2023-11-03T14:59:58.879-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Resolving credentials provider: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:47 @module=aws.aws-base tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ConfigureProvider timestamp=2023-11-03T14:59:58.879-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Using profile: tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 tf_aws.profile.source=provider tf_mux_provider=*schema.GRPCProviderServer tf_aws.profile=<redacted> tf_rpc=ConfigureProvider @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:47 @module=aws.aws-base timestamp=2023-11-03T14:59:58.879-0500
2023-11-03T14:59:58.879-0500 [DEBUG] provider.terraform-provider-aws_v5.24.0_x5: Loading profile: tf_mux_provider=*schema.GRPCProviderServer tf_rpc=ConfigureProvider @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:47 @module=aws.aws-base tf_aws.profile=<redacted> tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=d3aa1a9d-c926-1318-bc83-d230121b7eb7 timestamp=2023-11-03T14:59:58.879-0500

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None
@mire3212 mire3212 added the bug Addresses a defect in current functionality. label Nov 3, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 3, 2023

Community Note

Voting for Prioritization

  • Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

  • If you are interested in working on this issue, please leave a comment.
  • If this would be your first contribution, please review the contribution guide.

@terraform-aws-provider terraform-aws-provider bot added the needs-triage Waiting for first response or review from a maintainer. label Nov 3, 2023
@ewbankkit
Copy link
Contributor

ewbankkit commented Nov 3, 2023 •
edited
Loading

@mire3212 Thanks for raising this issue 👏.
Could you please paste in your provider "aws" {...} configuration (redacting anything sensitive)?
In particular, are you using an external AWS config or credentials file?

@ewbankkit ewbankkit added waiting-response Maintainers are waiting on response from community or contributor. and removed needs-triage Waiting for first response or review from a maintainer. labels Nov 3, 2023
@corrylc
Copy link

corrylc commented Nov 3, 2023 •
edited
Loading

I am also seeing this issue with 5.24. Downgrading to v5.23.1 resolved the issue.

The plan does eventually compute, but it takes many minutes for Terraform to even start the planning process.

aws = {
      source  = "hashicorp/aws"
      version = ">= 5.3.0"
}

In my case, this is with AWS SSO credentials loading from a profile in config

You can see the fantastically slow credential load in the log below, on the last line, where the time delta is enormous:

TF_LOG=INFO  terragrunt plan
2023-11-03T14:09:52.927-0700 [INFO]  Terraform version: 1.5.6
2023-11-03T14:09:52.927-0700 [INFO]  Go runtime version: go1.20.7
2023-11-03T14:09:52.927-0700 [INFO]  CLI args: []string{"/opt/homebrew/Cellar/tfenv/3.0.0/versions/1.5.6/terraform", "plan"}
2023-11-03T14:09:52.927-0700 [INFO]  CLI command args: []string{"plan"}
2023-11-03T14:09:52.931-0700 [INFO]  Attempting to use session-derived credentials
2023-11-03T14:09:53.366-0700 [INFO]  Successfully derived credentials from session
2023-11-03T14:09:53.366-0700 [INFO]  AWS Auth provider used: "SSOProvider"
2023-11-03T14:09:53.922-0700 [INFO]  backend/local: starting Plan operation
2023-11-03T14:09:55.092-0700 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:09:57.104-0700 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:09:57.104-0700
2023-11-03T14:09:57.440-0700 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:09:57.576-0700 [INFO]  provider.terraform-provider-random_v3.5.1_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:09:57.576-0700
2023-11-03T14:09:57.597-0700 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:09:57.677-0700 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:09:57.677-0700
2023-11-03T14:09:57.993-0700 [INFO]  backend/local: plan calling Plan
2023-11-03T14:09:57.994-0700 [INFO]  provider: configuring client automatic mTLS
2023-11-03T14:09:58.073-0700 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: configuring server automatic mTLS: timestamp=2023-11-03T14:09:58.072-0700
2023-11-03T14:11:49.008-0700 [INFO]  provider.terraform-provider-aws_v5.24.0_x5: Retrieved credentials: @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:39 @module=aws.aws-base tf_provider_addr=registry.terraform.io/hashicorp/aws tf_aws.credentials_source=SSOProvider tf_mux_provider=*schema.GRPCProviderServer tf_req_id=22cf0b07-38a1-2e9b-097f-d3b684aadde4 tf_rpc=ConfigureProvider timestamp=2023-11-03T14:11:49.008-0700
Interrupt received.

@github-actions github-actions bot removed the waiting-response Maintainers are waiting on response from community or contributor. label Nov 3, 2023
@mire3212
Copy link
Author

mire3212 commented Nov 3, 2023

I've tested with both profile = <xyz> and by using the AWS environment variables.

terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5"
    }
  }
}

This specific configuration relies on the default profile, but like I mentioned, I've tested with a direct profile = <xyz> and get the same error.

@IAXES
Copy link

IAXES commented Nov 3, 2023

Seeing the same behavior. Running tf apply (Terragrunt latest) on Linux (x86_64) on a fairly powerful workstation took about 20 minutes to apply a "hello world" example (creating some basic IAM resources).

Quickfix for now, skipping v5.24.x series.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = ">= 5.22.0, != 5.24"
    }
  }
}

@ewbankkit
Copy link
Contributor

Relates #34234.

@terraform-aws-provider terraform-aws-provider bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Nov 5, 2023
@ewbankkit ewbankkit added provider Pertains to the provider itself, rather than any interaction with AWS. authentication Pertains to authentication; to the provider itself of otherwise. labels Nov 6, 2023
@SamuZad
Copy link

SamuZad commented Nov 7, 2023

Might relate to aws/aws-sdk-go-v2#2353

We have also experienced significantly downgraded performance (3 minute plans instead of 30 seconds previously), but downgrading to prior versions doesn't seem to improve things 😢

@ewbankkit
Copy link
Contributor

ewbankkit commented Nov 7, 2023 •
edited
Loading

aws/aws-sdk-go-v2#2355 fixes aws/aws-sdk-go-v2#2353 and will likely be available in today's AWS SDK for Go v2 release which we should merge for this week's v5.25.0 Terraform AWS Provider release.

@ewbankkit ewbankkit mentioned this issue Nov 7, 2023
Merged
@github-actions github-actions bot added this to the v5.25.0 milestone Nov 7, 2023
@github-actions github-actions bot removed the bug Addresses a defect in current functionality. label Nov 10, 2023
@github-actions GitHub Actions
Copy link

This functionality has been released in v5.25.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@lukehoban lukehoban mentioned this issue Nov 29, 2023
Closed
@github-actions GitHub Actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 11, 2023
@justinretzolk justinretzolk added the bug Addresses a defect in current functionality. label Feb 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@gdavison gdavison

Labels
authentication Pertains to authentication; to the provider itself of otherwise. bug Addresses a defect in current functionality. prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. provider Pertains to the provider itself, rather than any interaction with AWS.
Projects
None yet
Milestone
v5.25.0
Development

Successfully merging a pull request may close this issue.

Rolls back AWS SDK modules with shared config parsing bug hashicorp/terraform-provider-aws
7 participants
@mire3212 @gdavison @ewbankkit @IAXES @justinretzolk @SamuZad @corrylc