diff --git a/README.md b/README.md index 53ba6711..c7b0690c 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,14 @@ The Mentor building block enables effective mentoring interactions between mento
+## System Requirements + +- **Operating System:** Ubuntu 22 +- **Node.js:** v20 +- **PostgreSQL:** 16 +- **Citus:** 12.1 +- **Apache Kafka:** 3.5.0 + # Setup Options Elevate user services can be set in local using two methods: @@ -129,150 +137,517 @@ Elevate user services can be set in local using two methods: -
Local Service with local dependencies(Hardest) +
+ +Local Service with local dependencies(Hardest) **Expectation**: Run single service with existing local dependencies in host (**Non-Docker Implementation**). -### Steps +## Installations + +### Install Node.js LTS + +Refer to the [NodeSource distributions installation scripts](https://github.com/nodesource/distributions#installation-scripts) for Node.js installation. -1. Install required tools & dependencies +```bash +$ curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash - &&\ +sudo apt-get install -y nodejs +``` - Install any IDE (eg: VScode) +### Install Build Essential - Install Nodejs: https://nodejs.org/en/download/ +```bash +$ sudo apt-get install build-essential +``` - Install MongoDB: https://docs.mongodb.com/manual/installation/ +### Install Kafka - Install Robo-3T: ​​ https://robomongo.org/ +Refer to [Kafka Ubuntu 22.04 setup guide](https://www.fosstechnix.com/install-apache-kafka-on-ubuntu-22-04-lts/) -2. Clone the **User service** repository. +1. Install OpenJDK 11: + ```bash + $ sudo apt install openjdk-11-jdk ``` - git clone https://github.com/ELEVATE-Project/user.git + +2. Download and extract Kafka: + + ```bash + $ sudo wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgz + $ sudo tar xzf kafka_2.12-3.5.0.tgz + $ sudo mv kafka_2.12-3.5.0 /opt/kafka ``` -3. Add **.env** file to the project directory +3. Configure Zookeeper: + + ```bash + $ sudo nano /etc/systemd/system/zookeeper.service + ``` + + Paste the following lines into the `zookeeper.service` file: + + ```ini + /etc/systemd/system/zookeeper.service + [Unit] + Description=Apache Zookeeper service + Documentation=http://zookeeper.apache.org + Requires=network.target remote-fs.target + After=network.target remote-fs.target - Create a **.env** file in **src** directory of the project and copy these environment variables into it. + [Service] + Type=simple + ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties + ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh + Restart=on-abnormal + [Install] + WantedBy=multi-user.target ``` - #User Service Config - # Port on which service runs - APPLICATION_PORT = 3000 + Save and exit. - # Service environment - APPLICATION_ENV = development +4. Reload systemd: - # Database connectivity url - DATABASE_URL=postgres://postgres:postgres@localhost:5432/elevate-user + ```bash + $ sudo systemctl daemon-reload + ``` - # Token secret to generate access token - ACCESS_TOKEN_SECRET = 'access-token-secret' +5. Configure Kafka: - # Token secret to generate refresh token - REFRESH_TOKEN_SECRET = 'refresh-token-secret' + ```bash + $ sudo nano /etc/systemd/system/kafka.service + ``` - # Kafka hosted server url - KAFKA_URL = localhost:9092 + Paste the following lines into the `kafka.service` file: - # Kafka group to which consumer belongs - KAFKA_GROUP_ID = userservice + ```ini + [Unit] + Description=Apache Kafka Service + Documentation=http://kafka.apache.org/documentation.html + Requires=zookeeper.service - # Kafka topic to consume data from - KAFKA_TOPIC = 'topic' + [Service] + Type=simple + Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" + ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties + ExecStop=/opt/kafka/bin/kafka-server-stop.sh - # Kafka topic to push notification data - NOTIFICATION_KAFKA_TOPIC = notificationtopic + [Install] + WantedBy=multi-user.target + ``` - # Any one of three features available for cloud storage - CLOUD_STORAGE = 'GCP/AWS/AZURE' + Save and exit. - # Gcp json config file path - GCP_PATH = 'gcp.json' +6. Reload systemd: - # Gcp bucket name which stores files - DEFAULT_GCP_BUCKET_NAME = 'gcp-bucket-storage-name' + ```bash + $ sudo systemctl daemon-reload + ``` - # Gcp project id - GCP_PROJECT_ID = 'project-id' +7. Start Zookeeper: - # Aws access key id - AWS_ACCESS_KEY_ID = 'aws-access-key-id' + ```bash + $ sudo systemctl start zookeeper + ``` - # Aws secret access key - AWS_SECRET_ACCESS_KEY = 'aws-secret-access-key' + Check status: - # Aws region where bucket will be located - AWS_BUCKET_REGION = 'ap-south-1' + ```bash + $ sudo systemctl status zookeeper + ``` - # Aws end point - AWS_BUCKET_ENDPOINT = 's3.ap-south-1.amazonaws.com' + Zookeeper service status should be shown as active (running). - # Aws bucket name which stores files - DEFAULT_AWS_BUCKET_NAME = 'aws-bucket-storage-name' +8. Start Kafka: - # Azure storage account name - AZURE_ACCOUNT_NAME = 'account-name' + ```bash + $ sudo systemctl start kafka + ``` - # Azure storage account key - AZURE_ACCOUNT_KEY = 'azure-account-key' + Check status: + + ```bash + $ sudo systemctl status kafka + ``` - # Azure storage container which stores files - DEFAULT_AZURE_CONTAINER_NAME = 'azure-container-storage-name' + Kafka status should be shown as active (running). - # Internal access token for communication between services via network call - INTERNAL_ACCESS_TOKEN = 'internal-access-token' +### Install Redis + +Refer to [Redis Ubuntu 22.04 setup guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-22-04) + +1. Update the package list: + + ```bash + $ sudo apt update + ``` +2. Install Redis: - #Enable logging of network request - ENABLE_LOG = true + ```bash + $ sudo apt install redis-server + ``` - # JWT Access Token expiry In Days - ACCESS_TOKEN_EXPIRY = '1' +3. Configure Redis for systemd: - # JWT Refresh Token expiry In Days - REFRESH_TOKEN_EXPIRY = '183' + ```bash + $ sudo nano /etc/redis/redis.conf + ``` - # Redis Host connectivity url - REDIS_HOST = 'redis://localhost:6379' + Find the `supervised` directive and change it to "systemd" as follows: + + ```conf + . . . + # If you run Redis from upstart or systemd, Redis can interact with your + # supervision tree. Options: + # supervised no - no supervision interaction + # supervised upstart - signal upstart by putting Redis into SIGSTOP mode + # supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET + # supervised auto - detect upstart or systemd method based on + # UPSTART_JOB or NOTIFY_SOCKET environment variables + # Note: these supervision methods only signal "process is ready." + # They do not enable continuous liveness pings back to your supervisor. + supervised systemd + . . . + ``` - # Otp expiration time for forgetpassword or registration process - OTP_EXP_TIME = 86400 + Save and exit. - # Enable email based otp verification for registration process - ENABLE_EMAIL_OTP_VERIFICATION = true +4. Restart the Redis service: - # Api doc url - API_DOC_URL = '/api-doc' + ```bash + $ sudo systemctl restart redis.service ``` -4. Start MongoDB locally +### Install Single-Node Citus (Distributed Database) - Based on your host operating system and method used, start MongoDB. +Refer to [official Citus single-node setup](https://docs.citusdata.com/en/stable/installation/single_node_debian.html) -5. Install Npm packages +1. Download and install Citus: + ```bash + $ curl https://install.citusdata.com/community/deb.sh | sudo bash + $ sudo apt-get -y install postgresql-16-citus-12.1 ``` - ELEVATE/user/src$ npm install + +2. Switch to the PostgreSQL user: + + ```bash + $ sudo su - postgres ``` -6. Start User server +3. Set the PostgreSQL bin directory in the PATH and create a directory for Citus: + ```bash + $ export PATH=$PATH:/usr/lib/postgresql/16/bin + $ cd ~ + $ mkdir citus ``` - ELEVATE/user/src$ npm start + +4. Initialize the Citus database: + + ```bash + $ initdb -D citus ``` -
-
+5. Configure Citus in `citus/postgresql.conf`: + + ```bash + $ echo "shared_preload_libraries = 'citus'" >> citus/postgresql.conf + ``` + +6. Start the Citus server: + + ```bash + $ pg_ctl -D citus -o "-p 9700" -l citus_logfile start + ``` + +7. Create the Citus extension: + + ```bash + $ psql -p 9700 -c "CREATE EXTENSION citus;" + ``` + +8. Check the Citus version: + + ```bash + $ psql -p 9700 -c "select citus_version();" + ``` + + You should see an output similar to the following, indicating that Citus is successfully installed: + + ```sql + postgres=# select citus_version(); + citus_version + ---------------------------------------------------------------------------------------------------- + Citus 12.1.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit + (1 row) + ``` + +### Install PM2 + +Refer to [How To Set Up a Node.js Application for Production on Ubuntu 22.04](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-22-04). + +**Exit the postgres user account and run the following command** + +```bash +$ sudo npm install pm2@latest -g +``` + +## Setting up Repository + +### Clone the user repository to /opt/backend directory + +```bash +opt/backend$ git clone -b develop-2.5 --single-branch "https://github.com/ELEVATE-Project/user.git" +``` + +### Install Npm packages from src directory + +```bash +backend/user/src$ sudo npm i +``` + +### Create .env file in src directory + +```bash +user/src$ sudo nano .env +``` + +Copy-paste the following env variables to the `.env` file: + +NB : Make sure to update the credentials according to your configurations. + +```env +ACCESS_TOKEN_EXPIRY= 10 +ACCESS_TOKEN_SECRET= asadsd8as7df9as8df987asdf +ADMIN_INVITEE_UPLOAD_EMAIL_TEMPLATE_CODE= invitee_upload_status +ADMIN_SECRET_CODE= Na7ad23ws5cm3kfmw24dmdsflaksd +API_DOC_URL=/user/api-doc +APPLICATION_ENV=development +APPLICATION_PORT=3001 +APP_NAME=MentorED + +AWS_ACCESS_KEY_ID= "adsfg98a7sdfg" +AWS_BUCKET_ENDPOINT="s3.ap-south-1.amazonaws.com" +AWS_BUCKET_REGION="ap-south-1" +AWS_SECRET_ACCESS_KEY="asd9786fg9a8sd/asdfg9a8sd7fg" + + +AZURE_ACCOUNT_KEY=asd897gfa09sd87f09as8d +AZURE_ACCOUNT_NAME=mentoring +CLEAR_INTERNAL_CACHE=userinternal +CLOUD_STORAGE= GCP +DEFAULT_AWS_BUCKET_NAME=mentoring-dev-storage +DEFAULT_AZURE_CONTAINER_NAME=mentoring-images +DEFAULT_GCP_BUCKET_NAME=mentoring-dev-storage + +DEFAULT_ORGANISATION_CODE= default_code +DEFAULT_ORG_ID= 1 +DEFAULT_QUEUE= user-queue +DEFAULT_ROLE= mentee +DEV_DATABASE_URL= postgres://shikshalokam:slpassword123@localhost:9700/elevate_user +DISABLE_LOG= false +EMAIL_ID_ENCRYPTION_ALGORITHM= aes-256-cbc +EMAIL_ID_ENCRYPTION_IV= a19f1ewaqwei9e03edkc32e +EMAIL_ID_ENCRYPTION_KEY= 9bszawjkckw2e3dm35fcw27ws4ed5rftg6y6y7y7654tf4rwq5tr0ol2qa9owsie +ENABLE_EMAIL_OTP_VERIFICATION=true +ENABLE_LOG=true +ERROR_LOG_LEVEL=silly +EVENT_ENABLE_ORG_EVENTS=true +EVENT_ORG_LISTENER_URLS=http://localhost:3567/mentoring/v1/organization/eventListener +GCP_PATH=gcp.json +GCP_PROJECT_ID=sl-dev-project +GENERIC_INVITATION_EMAIL_TEMPLATE_CODE=generic_invite +INTERNAL_ACCESS_TOKEN= Fqdkfaswekdlwe +INTERNAL_CACHE_EXP_TIME= 86400 +INVITEE_EMAIL_TEMPLATE_CODE= invite_user +IV= LKYTTAqkajswiawqw/Z== +KAFKA_GROUP_ID=dev.users +KAFKA_TOPIC= dev.topic +KAFKA_URL= localhost:9092 +KEY= W/m2cr/aMswjrdsa23sgfy5e34d+bKcbAWZSLjJP2qY= +MENTEE_INVITATION_EMAIL_TEMPLATE_CODE= invite_mentee +MENTORING_SERVICE_URL= http://localhost:3000 +MENTOR_INVITATION_EMAIL_TEMPLATE_CODE= invite_mentor +MENTOR_REQUEST_ACCEPTED_EMAIL_TEMPLATE_CODE= mentor_request_accepted +MENTOR_REQUEST_REJECTED_EMAIL_TEMPLATE_CODE= mentor_request_rejected +MENTOR_SECRET_CODE=4567 +NOTIFICATION_KAFKA_TOPIC=dev.notification +ORG_ADMIN_INVITATION_EMAIL_TEMPLATE_CODE= invite_org_admin +OTP_EMAIL_TEMPLATE_CODE= emailotp +OTP_EXP_TIME= 86400 +PORTAL_URL= "https://dev.elevate-mentoring.shikshalokam.org/auth/login" +RATING_KAFKA_TOPIC= dev.mentor_rating +REDIS_HOST= redis://localhost:6379 +REFRESH_TOKEN_EXPIRY= 183 +REFRESH_TOKEN_SECRET=371hkjadidy2ashiKAkajshdkid23iuekw71yekiaskdvkvegxvy23t78veQwexqviveit6ttZyeeytx62tx236uv +REFRESH_VIEW_INTERVAL=30000 +REGISTRATION_EMAIL_TEMPLATE_CODE= registration +REGISTRATION_OTP_EMAIL_TEMPLATE_CODE= registrationotp +SALT_ROUNDS= 10 +SAMPLE_CSV_FILE_PATH= sample/bulk_user_creation.csv +SCHEDULER_SERVICE_BASE_URL= /scheduler/ +SCHEDULER_SERVICE_ERROR_REPORTING_EMAIL_ID= rakesh.k@pacewisdom.com +SCHEDULER_SERVICE_HOST= http://localhost:3567 +SCHEDULER_SERVICE_URL= http://localhost:3567/jobs/scheduleJob +created_time= 2024-02-08T07:40:04.571464939Z +custom_metadata= null +destroyed= false +version= 31 + +``` + +Save and exit. -# Tech stack +## Setting up Databases -- Node - 16.0.0 -- Kafka - 3.1.0 -- Jest - 28.1.1 -- MongoDB - 4.1.4 -- Redis - 7.0.0 +**Log into the postgres user** + +```bash +sudo su postgres +``` + +**Log into psql** + +```bash +psql -p 9700 +``` + +**Create a database user/role:** + +```sql +CREATE USER shikshalokam WITH ENCRYPTED PASSWORD 'slpassword'; +``` + +**Create the elevate_user database** + +```sql +CREATE DATABASE elevate_user; +GRANT ALL PRIVILEGES ON DATABASE elevate_user TO shikshalokam; +\c elevate_user +GRANT ALL ON SCHEMA public TO shikshalokam; +``` + +## Running Migrations To Create Tables + +**Exit the postgres user account and install sequelize-cli globally** + +```bash +$ sudo npm i sequelize-cli -g +``` + +**Navigate to the src folder of user service and run sequelize-cli migration command:** + +```bash +user/src$ npx sequelize-cli db:migrate +``` + +**Now all the tables must be available in the Citus databases** + +## Setting up Distribution Columns in Citus PostgreSQL Database + +Refer [Choosing Distribution Column](https://docs.citusdata.com/en/stable/sharding/data_modeling.html) for more information regarding Citus distribution columns. + +**Login into the postgres user** + +```bash +sudo su postgres +``` + +**Login to psql** + +```bash +psql -p 9700 +``` + +**Login to the elevate_user database** + +```sql +\c elevate_user +``` + +**Enable Citus for elevate_user** + +```sql +CREATE EXTENSION citus; +``` + +**Within elevate_user, run the following queries:** + +```sql +SELECT create_distributed_table('entities', 'entity_type_id'); +SELECT create_distributed_table('entity_types', 'organization_id'); +SELECT create_distributed_table('file_uploads', 'organization_id'); +SELECT create_distributed_table('forms', 'organization_id'); +SELECT create_distributed_table('notification_templates', 'organization_id'); +SELECT create_distributed_table('organizations', 'id'); +SELECT create_distributed_table('organization_codes', 'code'); +SELECT create_distributed_table('organization_domains', 'domain'); +SELECT create_distributed_table('organization_role_requests','organization_id'); +SELECT create_distributed_table('organization_user_invites','organization_id'); +SELECT create_distributed_table('users_credentials','email'); +SELECT create_distributed_table('users', 'organization_id'); +``` + +## Running Seeder to Populate the Tables with Seed Data + +**Exit the postgres user navigate to the script folder of the user service** + +**Run the insertDefaultOrg.js script** + +```bash +src/scripts$ node insertDefaultOrg.js +``` + +_Keep note of the default organization id generated by the script_ + +**Navigate to the src folder of the user service and update the .env file with these variables:** + +```env +DEFAULT_ORG_ID= +DEFAULT_ORGANISATION_CODE=default_code +``` + +**Run the seeder command** + +```bash +src$ npm run db:seed:all +``` + +## Start the Service + +Run pm2 start command: + +```bash +user/src$ pm2 start app.js -i 2 --name elevate-user +``` + +#### Run pm2 ls command + +```bash +$ pm2 ls +``` + +Output should look like this (Sample output, might slightly differ in your installation): + +```bash +┌────┬─────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐ +│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │ +├────┼─────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤ +│ 1 │ elevate-user │ default │ 1.0.0 │ cluster │ 106976 │ 27h │ 0 │ online │ 0% │ 167.0mb │ jenkins │ disabled │ +│ 2 │ elevate-user │ default │ 1.0.0 │ cluster │ 106986 │ 27h │ 0 │ online │ 0% │ 169.3mb │ jenkins │ disabled │ +└────┴─────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘ +``` + +This concludes the services and dependency setup. + +## Postman Collections + +- [User Service](https://github.com/ELEVATE-Project/user/tree/develop-2.5/src/api-doc) + +
+
# Migrations Commands