Github Action to Deploy Talawa-API to GoDaddy VPS

[vasujain275]

What kind of change does this PR introduce?

It is a feature, this pr introduces deploy.yml file which can be used to trigger github actions on every push to deploy branch that pushes talawa-api to ec2 instance.

Issue Number:

Fixes #1428

Did you add tests for your changes?

N/A

Snapshots/Videos:
This snapshot is from my fork of talawa-api that shows github action working correctly.(https://github.com/vasujain275/talawa-api/actions/runs/7185962040)

If relevant, did you update the documentation?

Summary

Solves this issue - #1428

Does this PR introduce a breaking change?
No

Other information

So, this solution as you can see deploy the whole talwa-api on aws free ec2 in ~15mins which is alot. I first run this commands on fresh ec2 machine

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install curl
sudo curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash
source ~/.bashrc
nvm install --lts
sudo apt-get install docker.io -y

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose
git clone https://github.com/PalisadoesFoundation/talawa-api.git
cd talawa-api
npm install
npm run setup

Then the action can deploy talawa-api using action secrets which are HOST that is IP of ec2 instance, USERNAME that is ubuntu for our ubuntu ec2, PORT that is 22 for sshing and private key that we need to generate on our ec2 first.

Have you read the contributing guide?
Yes

@palisadoes @kb-0311 @noman2002 any suggestion for how to reduce build time? I am actively looking for a better solution and will commit the changes once I found a better solution.

[github-actions]

Our Pull Request Approval Process

We have these basic policies to make the approval process smoother for our volunteer team.

Testing Your Code

Please make sure your code passes all tests. Our test code coverage system will fail if these conditions occur:

1. The overall code coverage drops below the target threshold of the repository
2. Any file in the pull request has code coverage levels below the repository threshold
3. Merge conflicts

The process helps maintain the overall reliability of the code base and is a prerequisite for getting your PR approved. Assigned reviewers regularly review the PR queue and tend to focus on PRs that are passing.

Reviewers

When your PR has been assigned reviewers contact them to get your code reviewed and approved via:

1. comments in this PR or
2. our slack channel

Reviewing Your Code

Your reviewer(s) will have the following roles:

1. arbitrators of future discussions with other contributors about the validity of your changes
2. point of contact for evaluating the validity of your work
3. person who verifies matching issues by others that should be closed.
4. person who gives general guidance in fixing your tests

CONTRIBUTING.md

Read our CONTRIBUTING.md file. Most importantly:

1. PRs with issues not assigned to you will be closed by the reviewer
2. Fix the first comment in the PR so that each issue listed automatically closes

Other

1. 🎯 Please be considerate of our volunteers' time. Contacting the person who assigned the reviewers is not advised unless they ask for your input. Do not @ the person who did the assignment otherwise.
2. Read the CONTRIBUTING.md file make

[palisadoes]

Please answer my question on the GoDaddy VPS.

• Cloud Based API Instance for Developers #1428

With that solution we would have a fixed cost and not have to worry about unexpected cloud charges.

Let me know

[palisadoes]

#1428

Please answer this question. The EC2 solution is premature. The contributors feel it's high risk

[vasujain275]

@palisadoes

1. Got access to demo vps using ssh.
2. I am currently looking into optimizing the build times, once done I will commit the changes to this pr.

[kb-0311]

@vasujain275 @palisadoes Here are my thoughts.

1. It's okay if you are not able to optimize the build time, I suggest that it can be looked at later the priority should be to implement the deployment pipeline which is most suitable first with the main branch. I guess the main bottle neck is during pruning and restarting the docker services during build time on push.

I first run this commands on fresh ec2 machine

For now, I will just collectively use the term VM to refer to ec2 or a vps.
If I am not wrong our VPS or ec2 instance will have persistent storage so those command sequences you listed in the original issue will only be needed once initially just to configure our VM, correct me if I am wrong.
3. You have not mentioned how pulling the latest main branch from the talawa-api/repo dir will be handled. Right now in your workflow file on every push, we will be executing remote commands to pull the latest branch and restart the docker services when this workflow is triggered. What happens if a particular command fails (like a connection time out error during pulling a docker image while running compose )? How do you think we can handle.
4. Regarding the roles part, I agree with @vasujain275 approach let the all the manual routine checkups and introspection be made available to unprivileged accounts but let the IAM sudo user have the privilege to modify the services/packages.

Also if @palisadoes @vasujain275 please do create a channel and add me, I think we should have a small meet regarding this when we will be free.

[palisadoes]

The #talawa-workflows slack channel has been created

[vasujain275]

@kb-0311

1. I will try to optimize the pruning and restarting of the docker services, otherwise as you suggested, I will focus on basic implementation for now.
2. Yes commands listed are only required to run once when setting up the VM for the first time.
3. The action first pulls the latest branch and builds containers again but all errors are not handled right now. However, according to me the actions can't have a connection timeout error as I have mentioned a connection time of 200min, the only error I can think of this error in building the docker image of talawa-api which is I think prevented by our tests. Correct me if I am wrong.
4. I got ssh access to the IAM sudo user on the demo VPS and will get that up and running in a few hours.

@palisadoes, As @kb-0311 suggested a small meeting regarding this would be helpful.

[vasujain275]

@kb-0311 @palisadoes
Multiple changes are required to this PR like the addition of MongoDB reset script, redis rate limiter, etc as discussed in the meeting. How do you think I should proceed with this? Make different PRs for each of them or include them here only?

[palisadoes]

Make separate PRs for each and link them to:

• Cloud Based API Instance for Developers #1428

[kb-0311]

@vasujain275 Yes there should be different PRs for both of those tasks. You can make an issue for now and tag me as reviewer.

[vasujain275]

@vasujain275 Yes there should be different PRs for both of those tasks. You can make an issue for now and tag me as reviewer.

Got it, making different prs for them

[codecov]

Codecov Report

Attention: 91 lines in your changes are missing coverage. Please review.

Comparison is base (c0468a4) 98.17% compared to head (d32d6ab) 97.45%.
Report is 123 commits behind head on develop.

Files	Patch %	Lines
src/utilities/PII/decryption.ts	0.00%	17 Missing ⚠️
src/resolvers/Mutation/updateUserProfile.ts	70.90%	1 Missing and 15 partials ⚠️
src/resolvers/middleware/currentUserExists.ts	45.83%	13 Missing ⚠️
src/utilities/PII/encryption.ts	0.00%	13 Missing ⚠️
src/utilities/PII/isAuthorised.ts	0.00%	11 Missing ⚠️
...tilities/encodedVideoStorage/uploadEncodedVideo.ts	90.12%	8 Missing ⚠️
src/resolvers/Mutation/createPost.ts	86.95%	6 Missing ⚠️
src/resolvers/Mutation/removeAdvertisement.ts	92.85%	2 Missing ⚠️
...c/resolvers/Query/postsByOrganizationConnection.ts	33.33%	2 Missing ⚠️
src/resolvers/Subscription/onPluginUpdate.ts	0.00%	2 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1491      +/-   ##
===========================================
- Coverage    98.17%   97.45%   -0.73%     
===========================================
  Files          184      214      +30     
  Lines        10767    12949    +2182     
  Branches       835     1042     +207     
===========================================
+ Hits         10571    12620    +2049     
- Misses         186      299     +113     
- Partials        10       30      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vasujain275]

1. The issue is that setup.ts is going to change soon. So you'll need to let the person who is working on the MongoDB automation know your requirements for deployment.
2. We don't want this to break within a week of implementation

Understood, reverted all the changes to setup.ts and informed the person working on this about my requirements for deployment here - #1605 (comment)

[palisadoes]

@vasujain275

1. In order to get this PR closed and the cloud API instance automation, restore your changes to setup.ts. There is no guarantee that the other issue will be finished in time.
2. With those restored changes will this PR be ready to implement the cloud API instance?
3. Is anything else necessary?

[kb-0311]

@vasujain275 @palisadoes I think we keep Vasu's changes to the setup.ts as is.

I have reviewed this PR and the scripts are fine.

[palisadoes]

Kanishka, should we merge this now?

[vasujain275]

@palisadoes @kb-0311

1. The changes in setup.ts are not needed for the functioning of the cloud instance.
2. They will be needed to setup cloud instance easily next time.
3. The pr is not ready to merge as I have to add instructions to for creating cronjobs in documentation.
4. I will implement them in 4-5 hours today itself, then the pr will be ready to merge.

So, we can wait for the changes in setup.ts to be implemented in #1605 as it doesn't directly affect the cloud instance -

[kb-0311]

@vasujain275 Noted. Do let me know when then PR is ready for review

[vasujain275]

@kb-0311
All scripts, containers,cronjobs are working on vps and well documented for anyone to replicate. But I have a doubt - I haven't configured RECAPTCHA_SECERT_KEY, MAIL_USERNAME, MAIL_PASSWORD as they require a google account to generate all of these, how should I add them?

[kb-0311]

@vasujain275 cool I will review everything tomorrow.

I have a doubt - I haven't configured RECAPTCHA_SECERT_KEY, MAIL_USERNAME, MAIL_PASSWORD as they require a google account to generate all of these, how should I add them

@palisadoes any suggestions ?

[palisadoes]

@kb-0311 I have no further comments. Please take a look

[palisadoes]

See comment

[vasujain275]

I have a doubt - I haven't configured RECAPTCHA_SECERT_KEY, MAIL_USERNAME, MAIL_PASSWORD as they require a google account to generate all of these, how should I add them

@kb-0311 any update?

[kb-0311]

@vasujain275 sorry for the late reply.

1. You can create a dummy google account for now and share it with @palisadoes. You can use this account for the env variables. Once we have made sure that the pipeline is working fine, we can change the account and those variables anytime.
2. While generating the recaptha secret key v3 make sure to save the corresponding site key somewhere we will need it to configure the clients apps that will be accessing the deployed API.

@palisadoes anything else you would like to add?

[vasujain275]

@kb-0311 Created the demo account and added the required recaptcha parms to .env and sent all the details to @palisadoes

[kb-0311]

Cool then please do take a look at my review, @palisadoes anything you want to add?

[palisadoes]

Cool then please do take a look at my review, @palisadoes anything you want to add?

1. I sent the Palisadoes keys to @vasujain275
2. @vasujain275 where is the .env file in this PR? Will it be autogenerated? If so, we should store the captcha keys as repo secrets for the best security

[vasujain275]

@palisadoes

1. Added secret keys to .env file on vps.
2. The .env file is stored in /home/talawa-api/develop/talawa-api/ directory on the Godaddy VPS. It will not be autogenerated and needed to be manually set by the person setting up the vps as mentioned in /scripts/cloud-api-demo/README.md.
3. Yes it will be best to store the captcha keys as repo secrets for the best security.