Update Terraform scripts to work with latest Terraform AWS provider and RDS module

[pli888]

Pull request for issue: #995 terraform plan causes unsupported argument errors

This is a pull request for the following functionalities:

• Terraform scripts now able to work with latest version 4.8.0 of Terraform AWS provider and version 4.2.0 of Terraform AWS RDS module.

How to test the new feature

1. Go to /gigadb-website/ops/infrastructure/envs/staging environment directory and ensure that it contains only the .gitignore file.
2. Execute ../../../scripts/tf_init.sh --project gigascience/forks/<YOUR_GITHUB_USERNAME>-gigadb-website --env staging. This should download the latest version 4.8.0 AWS provider for Terraform.
3. Run terraform plan
4. Run terraform apply - you should have the required permissions to create the AWS resources because there is a new IAM policy RDSPolicy_PR1001 containing the contents of docs/awsdocs/policy-rds.md in this PR which I have already attached to your AWS user accounts.
5. Run terraform refresh.
6. Run ../../../scripts/ansible_init.sh --env staging
7. Run ansible-playbook -i ../../inventories bastion_playbook.yml
8. Run TF_KEY_NAME=private_ip ansible-playbook -i ../../inventories dockerhost_playbook.yml
9. Test cannot access ec2_dockerhost instance from localhost:

$ ssh -i "~/.ssh/<FILENAME>.pem" centos@ec2-<IP_ADDRESS>.<AWS_REGION>.compute.amazonaws.com

10. Test can access bastion instance:

$  ssh -i "~/.ssh/<FILENAME>.pem" centos@ec2-<IP_ADDRESS>.<AWS_REGION>.compute.amazonaws.com
The authenticity of host 'ec2-xx-xxx-xx-xx.ap-northeast-1.compute.amazonaws.com (xx.xxx.xx.xx)' can't be established.
ECDSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'ec2-xx-xxx-xx-xx.ap-northeast-1.compute.amazonaws.com' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu Mar 31 05:41:55 2022 from xx.xxx.xx.xx
[centos@ip-10-99-0-17 ~]$

11. Test can access ec2_dockerhost instance from localhost via bastion:

# Configure SSH agent
$ ssh-add -K <FILENAME>.pem
# Connect to bastion
$ ssh -A centos@ec2-<IP_ADDRESS>.ap-northeast-1.compute.amazonaws.com
Identity added: /Users/xxxxxxx/.ssh/<FILENAME>.pem (/Users/xxxxxxx/.ssh/<FILENAME>.pem)
# Connect to staging server via private address
$ ssh centos@<ec2_dockerhost_private_ip>
The authenticity of host '10.99.0.189 (10.99.0.189)' can't be established.
ECDSA key fingerprint is SHA256: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.99.0.189' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Thu Mar 31 05:50:11 2022 from 10.99.0.17
[centos@ip-10-99-0-189 ~]$ 

12. Test can access RDS instance from localhost via bastion:

[centos@ip-10-99-0-189 ~]$ exit
[centos@ip-10-99-0-17 ~]$ export PGPASSWORD='<gigadb_db_password>'; psql -U gigadb -d postgres -h <gigadb_db_host> -p 5432
psql (11.15, server 11.13)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> 

13. Check sd_deploy step on your CI/CD pipeline is able to deploy against the AWS infrastructure.

Code Changes

Changes to the provisioning

• policy-rds.md has been updated to move CreateDBParameterGroup and ModifyDBParameterGroup permissions into the CreateResourcesforRDSInstances section. Originally, the permission was in the CreateRDSInstancesWithRegionAndInstanceTypeRestriction section but this now stops the DB parameter group from being created. There are other cosmetic changes in policy-rds.md.
• ops/infrastructure/modules/rds-instance/input.tf has been updated to include vpc_database_subnet_group variable which is now required for instantiating an RDS instance when using the latest version of Terraform AWS module.
• rds-instance.tf has been updated to work with the latest Terraform AWS module version. The variables name and final_snapshot_identifier are now called db_name and final_snapshot_identifier_prefix, respectively. In addition, the aws_db_parameter_group.gigadb-db-param-group resource has a name which has the AWS owner's name appended to it. In addition, the parameter_group_name in the rds module is provided using a reference to the name for the aws_db_parameter_group.gigadb-db-param-group resource.
• Using the latest Terraform AWs module appears to have fix the Error deleting DB parameter group: InvalidDBParameterGroupState: One or more database instances are still members error message which I don't see anymore when executing terraform destroy.
• ops/infrastructure/terraform.tf has been updated:
  • The name of the VPC has the aws region interpolated into it.
  • The database subnet is now created and the RDS is placed into it. There is no public access to the database subnet.
• Changes in terraform-inventory.sh and ansible_init.sh were required to include ./ at the end of terraform-inventory.sh calls to provide the location of the .tfstate file - see here for details.

[rija]

Hi @pli888,

I was able to perform all the provisioning steps and they all worked fine.

I was able to ssh the bastion the usual way I do with no problems:

$ ssh -i *************.pem [email protected]

and I could ssh the webapp ec2 instance the usual way I do without problems:

$ ssh -i *************.pem -o ProxyCommand='ssh -i *************.pem -vvv -W %h:%p -q [email protected]' [email protected]

All the tests are passing locally and on CI, and the staging deployment sd_deploy was successful and the website on my staging environment and
the corresponding portainer dashboard seem to work fine.

Finally terraform destroy did its job without errors.

I'm therefore happy to approve this PR.

Probably a stupid question:
In aws_db_parameter_group.gigadb-db-param-group[0].name, why is there [0]? I would think the DB parameter group resource is one thing, not an array of things.

[pli888]

Probably a stupid question:
In aws_db_parameter_group.gigadb-db-param-group[0].name, why is there [0]? I would think the DB parameter group resource is one thing, not an array of things.

It's because we use the count meta-argument as a conditional to create the aws_db_parameter_group resource when we are creating a staging environment. The problem is that the count value will also create that number of resources, hence we need a [0] in aws_db_parameter_group.gigadb-db-param-group[0].name to identify which aws_db_parameter_group.gigadb-db-param-group we want to use.

[kencho51]

Hi @pli888,

I have gone through all the steps mentioned in How to test the new feature and everything is working well as expected.

The gigadb_db_host is accessible on bastion server, and dockerhost server is accessible through the bastion server using ProxyCommadn from local dev computer.

And I have tried several rounds of terraform apply and terraform apply --destroy, DB parameter group can be created and destroyed successfully.

A suggestion is to move create_database_subnet_group = true in terraform.tf to above the comment block, so it would make the content below it more consistent. It is just a minor, cosmetic change.

Other than that, all code looks good to me.