provider should retry connection on different API address

[shundezhang]

Description

My juju controller has two IPs, a normal one and a fan IP.

api-endpoints: ['10.149.105.177:17070', '252.98.0.1:17070']

However, on the client machine where I run terraform command, the fan network is not accessible.
It seems the provider will pick one from the endpoint list randomly.
So sometimes it can talk to the controller, but sometimes it will return an error.

│ Error: Client Error
│ 
│   with juju_integration.kubernetes-control-plane-worker,
│   on main.tf line 325, in resource "juju_integration" "kubernetes-control-plane-worker":
│  325: resource "juju_integration" "kubernetes-control-plane-worker" {
│ 
│ dial tcp 252.98.0.1:17070: i/o timeout

It would be good if the provider can pick another address automatically rather than throwing an error or remember this somewhere that one address is not accessible and avoid it in the future.

Urgency

Annoying bug in our test suite

Terraform Juju Provider version

0.13.0

Terraform version

1.9.5

Juju version

3.5.2

Terraform Configuration(s)

No response

Reproduce / Test

terraform apply

Debug/Panic Output

No response

Notes & References

No response

[Aflynn50]

Hey @shundezhang, thanks for letting us know. This could actually be a juju bug since to connect to the api we actually call through to some juju code. This will require further investigation on our part.

[NucciTheBoss]

We have also encountered this problem trying to deploy the Slurm charms on AWS using the Juju Terraform provider. The provider is grabbing an internal IP address that cannot reach some of the machines deployed on AWS, so terraform apply fails. Some applications make it through, so it seems that our plan is using different IP addresses provided by Juju for communicating with the controller.