Update Docker dev. env.; support manual renewals of ICAs; outsource r…

…enewal survey to Google Forms; add GitHub templates (#638)

* Create venv in Docker images and add its binaries to beginning of PATH; replace apt-get w/ apt

* Mount only needed directories into /app, so that /app/venv is not wiped out

* Change Docker dev env OS to Rocky 8.8; use Python 3.10

* Restore WORKDIR in Postgres shell Dockerfile

* Install PostgreSQL 15 in DB shell image

* Correct base image for DB shell image

* Remove testing CMD

* Note expected possible error during DB load

* proof-of-concept google forms renewal survey

* remove original survey from renewal form

* Edit view and instructions

* add hard-coded json data for surveys

* un-hardcode check and block

* proof-of-concept google forms renewal survey

* remove original survey from renewal form

* Edit view and instructions

* add hard-coded json data for surveys

* un-hardcode check and block

* documentation

* fix tests, remove obsolete

* documentation

* fix tests, remove obsolete

* basic way for admins to review survey responses through renewal request view

* basic way for admins to review survey responses through renewal request view

* fix issues with tests

* fix issues with tests

* fix tests issues with tmp/credentials

* fix tests issues with tmp/credentials

* fix view for approval

* proof of concept export of existing responses ot google sheet

* fix view for approval

* proof of concept export of existing responses ot google sheet

* properly convert answers of existing responses, make the export function a command

* properly convert answers of existing responses, make the export function a command

* documentation

* documentation

* generate google form url in backend rather than in frontend

* generate google form url in backend rather than in frontend

* change file path to be in settings

* change file path to be in settings

* small fixes

* basic dummy backend

* dummy backend for local/development, actual validation for production

* small fixes

* basic dummy backend

* dummy backend for local/development, actual validation for production

* Method for renewing ICAs

* disallow non ICA projects

* Find PI

* Ensure requester is a part of the project. Stop requests for out of date allocation periods.

* Remove todo

* changes based off pr comments

* Initial unit testing

* Basic unit testing for key assumptions

* Refactor logic for fetching the AllocationPeriods pertinent to a particular allowance type into a method

* Refactor logic for setting PI eligibility in renewal request into method

* Refactor logic for computing SUs to allocate to new project request/renewal request into method

* Generalize projects command > renew to enable renewals of non-pooling FCAs and PCAs; add constraints

* Add comments describing additional test cases for projects command > renew

* flesh out testing for project renewals

* Disallow renewals of limited-per-PI allowances until checks added; add/fix tests

* Decorate renewal tests as being BRC tests

* Remove entry from .gitignore

* Replace 'docker-compose' binary w/ 'docker compose' in dev. scripts + README given Docker Desktop v4.32.0

* refactoring

* enforce dummy backend as default unless otherwise specified in overrides.yml

* renaming

* refactoring

* enforce dummy backend as default unless otherwise specified in overrides.yml

* renaming

* change survey data column index from number to letter

* change survey data column index from number to letter

* refactoring and renaming

* add get_survey_url to interface

* refactoring

* refactoring and renaming

* add get_survey_url to interface

* refactoring

* refactoring for interface

* refactor set_necessary_data

* Update approval_views.py

* documentation

* refactoring for interface

* refactor set_necessary_data

* documentation

* rename variables so not backend specific

* adjust interface

* Change Docker dev env OS to Rocky 8.8; use Python 3.10

* Restore WORKDIR in Postgres shell Dockerfile

* Install PostgreSQL 15 in DB shell image

* Correct base image for DB shell image

* Remove testing CMD

* Note expected possible error during DB load

* Always show survey step, LRC 2023-2024 survey added

* rename variables so not backend specific

* adjust interface

* Always show survey step, LRC 2023-2024 survey added

* update backend

* Adds PR, Bug and Feature issues templates

closes #634

* fix some issues

* Rename old ProjectRenewalSurveyForm to DeprecatedProjectRenewalSurveyForm; fix export data tests and command

* update backend

* fix some issues

* Rename old ProjectRenewalSurveyForm to DeprecatedProjectRenewalSurveyForm; fix export data tests and command

* generalize interface

* generalize interface

* move renewal_survey_utils to google backend with private methods

* refactor form and reword html

* unhardcode credentials path

* raise Exception in google backend and ValidationError in forms

* move renewal_survey_utils to google backend with private methods

* refactor form and reword html

* unhardcode credentials path

* raise Exception in google backend and ValidationError in forms

* scan google sheets for response in reverse order for latency

* permissive if Google sheets error

* rename google backend to google_forms

* scan google sheets for response in reverse order for latency

* permissive if Google sheets error

* rename google backend to google_forms

* Move renewal survey-related settings into a dict configured via YML files

* Add sub-sections to main.copyme to improve readability; remove unneeded entry from .gitignore

* Describe the return value of condition_dict methods of wizard views

* Add feature flag that allows renewal survey to be disabled

* Correct bug in fetching name for ComputingAllowance object

* fix mistakes

* Add new settings to testing config file

* check for flag in approval view

* clean up documentation for backend

* Install django-redis and configure it as cache backend; configure cache key for storing survey data

* Cache survey data to avoid repetitive hits to disk

* Replace dummy cache backend w/ Django one

* Add timeout of 24 hours to cached renewal survey data

* Remove hard-coded survey data; add sample file

* Rename module; change interface for validating completion; change returns values of permissive backend; update comments

* Move logic for computing renewal survey URL

* Simplify survey instructions

* Rename renewal survey module in test settings sample

* export data fix

* basic unit testing for ProjectRenewalSurveyForm

---------

Co-authored-by: Alastair Deng <[email protected]>
Co-authored-by: adeng27 <[email protected]>
Co-authored-by: helbashandy <[email protected]>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,33 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (Optional):**
+ - OS: [e.g. macOS, Windows]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+
+**Additional context (Optional)**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for a feature or enhancement
+title: ''
+labels: feature
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered (Optional)**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context (Optional)**
+Add any other context or screenshots about the feature request here.

.github/pull_request_template.md

@@ -0,0 +1,28 @@
+## Description
+
+**** Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ****
+
+Fixes #(issue number) 
+
+## Type of change
+
+ **** Please delete options that are not relevant. ****
+
+- Bug fix (non-breaking change which fixes an issue)
+- New feature (non-breaking change which adds functionality)
+- Breaking change (fix or feature that would cause existing functionality to not work as expected)
+
+## How Has This Been Tested?
+
+**** Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration. ****
+
+
+## PR Self Evaluation
+Strikethrough things that don’t make sense for your PR.
+
+- [ ] My code follows the agreed upon best practices
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding changes to the documentation (if needed)
+- [ ] My changes generate no new warnings
+- [ ] Any dependent changes have been merged and published in the appropriate modules
+- [ ] I have performed a self-review of my own code

.gitignore

@@ -39,4 +39,4 @@ google-drive-key.json
 overrides.yml
 New Project Request MOUs
 Secure Directory Request MOUs
-Service Units Purchase Request MOUs
+Service Units Purchase Request MOUs

bootstrap/ansible/main.copyme

@@ -24,6 +24,10 @@ enable_django_debug_toolbar: false
 # TODO: Generate one using: `openssl rand -base64 64`.
 django_secret_key:
 
+#------------------------------------------------------------------------------
+# PostgreSQL database settings
+#------------------------------------------------------------------------------
+
 # The name of the PostgreSQL database.
 # TODO: For LRC, set this to 'cf_lrc_db'.
 db_name: cf_brc_db
@@ -34,35 +38,53 @@ db_host: localhost
 db_admin_user: admin
 db_admin_passwd: ''
 
+#------------------------------------------------------------------------------
+# Redis settings
+#------------------------------------------------------------------------------
+
 # The password for Redis.
 # TODO: Replace the password.
 redis_passwd: ''
+
 redis_host: localhost
 
-# Log paths.
+#------------------------------------------------------------------------------
+# Logging settings
+#------------------------------------------------------------------------------
+
 # TODO: For LRC, use the substring 'cf_mylrc'.
 log_path: /var/log/user_portals/cf_mybrc
 portal_log_file: cf_mybrc_portal.log
 api_log_file: cf_mybrc_api.log
+
 logrotate_entry_name: cf_mybrc
+
 # TODO: Logs are only backed up if a path to an existent directory is given.
 log_backup_dir_path:
+
 # TODO: For Docker environments, set this to true.
 stream_logs_to_stdout: false
 
-# Apache settings.
+#------------------------------------------------------------------------------
+# Apache settings
+#------------------------------------------------------------------------------
+
 # The name of the copy of the generated WSGI template in the Apache directory.
 # TODO: For LRC, use the substring 'cf_mylrc'.
 wsgi_conf_file_name: cf_mybrc_wsgi.conf
 # TODO: For LRC, use the substring 'cf_lrc'.
 wsgi_conf_log_prefix: cf_brc
 
-# LRC Cloudflare settings.
+#------------------------------------------------------------------------------
+# LRC Cloudflare settings
+#------------------------------------------------------------------------------
+
 # Whether the web server is behind Cloudflare.
 # TODO: For the LRC production deployment, enable Cloudflare, since LBL
 # TODO: requires that web servers visible to the Internet be placed behind it.
 # TODO: https://commons.lbl.gov/display/cpp/Open+Web+Server+Requirements
 cloudflare_enabled: false
+
 # A list of Cloudflare's IP ranges.
 # TODO: Keep it up-to-date with: https://www.cloudflare.com/ips/.
 cloudflare_ip_ranges: [
@@ -82,51 +104,77 @@ cloudflare_ip_ranges: [
     197.234.240.0/22,
     198.41.128.0/17
 ]
+
 # The name of the server, which should differ from the name of the website.
 # Source: See Open Web Server Requirements link above.
 # TODO: Set this to e.g., mylrc-local.lbl.gov for mylrc.lbl.gov if Cloudflare
 # TODO: is enabled.
 cloudflare_local_server_name:
 
-# CILogon client settings.
+#------------------------------------------------------------------------------
+# CILogon client settings
+#------------------------------------------------------------------------------
+
 # TODO: Set these, needed only if SSO should be enabled.
 cilogon_app_client_id: ""
 cilogon_app_secret: ""
 
-# Django Flags settings.
+#------------------------------------------------------------------------------
+# Django Flags settings
+#------------------------------------------------------------------------------
+
+# # Note: Use uppercase True/False so that Python interprets these as booleans.
+
 # TODO: For LRC, disable link login.
 flag_basic_auth_enabled: False
 flag_sso_enabled: True
 flag_link_login_enabled: True
+
 # TODO: For LRC, disable BRC and enable LRC.
 flag_brc_enabled: True
 flag_lrc_enabled: False
+
 # The number of the month in which users should be able to request renewal for
 # the next allowance year.
 # TODO: For LRC, set the month number to 9 (September).
 flag_next_period_renewal_requestable_month: 5
+
+# Whether to enable UI support for users having multiple email addresses.
 flag_multiple_email_addresses_allowed: False
+
 # Whether to install and enable the MOU generation package.
 # TODO: For BRC, enable MOU generation (requires access to the package).
 flag_mou_generation_enabled: False
 
-# Portal settings.
+# Whether to include a survey as part of the allowance renewal request process.
+flag_renewal_survey_enabled: True
+
+#------------------------------------------------------------------------------
+# User-facing strings
+#------------------------------------------------------------------------------
+
 # TODO: For LRC, use "MyLRC", "Laboratory Research Computing", "LRC", and
 # TODO: "Lawrencium".
 portal_name: "MyBRC"
 program_name_long: "Berkeley Research Computing"
 program_name_short: "BRC"
+
 primary_cluster_name: "Savio"
+
 # TODO: For MyLRC, use "https://it.lbl.gov/service/scienceit/high-performance-computing/lrc/".
 center_user_guide: "https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/"
 # TODO: For MyLRC, use "https://it.lbl.gov/resource/hpc/for-users/getting-started/".
 center_login_guide: "https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/logging-brc-clusters/#Logging-in"
 # TODO: For MyLRC, use "[email protected]".
 center_help_email: "[email protected]"
 
-# # Storage settings.
-# # For more information, refer to the README.
+#------------------------------------------------------------------------------
+# BRC File storage settings
+#------------------------------------------------------------------------------
+
+# # Backend options: 'file_system', 'google_drive'
 # file_storage_backend: 'file_system'
+
 # new_project_request_mou_path: 'New Project Request MOUs/'
 # secure_directory_request_mou_path: 'Secure Directory Request MOUs/'
 # service_units_purchase_request_mou_path: 'Service Units Purchase Request MOUs/'
@@ -141,47 +189,84 @@ center_help_email: "[email protected]"
 # google_drive_private_key_file_path: '/path/to/google-drive-private-key.json'
 # google_drive_storage_media_root: '/'
 
-# # MOU generator settings.
-# # For more information, refer to the README.
+#------------------------------------------------------------------------------
+# BRC MOU generation settings
+#------------------------------------------------------------------------------
+
 # # TODO: For BRC deployments with access to the package for generating MOUs
 # # TODO: (e.g., production and staging), set the absolute path to the deploy
-# # TODO: key.
+# # TODO: key for the package.
 # mou_generator_deploy_key_path: '/path/to/id_mou_generator'
 
-# LRC billing validation package settings (for Pip installation).
+#------------------------------------------------------------------------------
+# LRC Billing validation settings
+#------------------------------------------------------------------------------
+
 # TODO: For LRC deployments with access to the package for billing validation
 # TODO: (e.g., production and staging), set these.
 install_billing_validation_package: false
 # Example: "gitlab.com/user/repo_name.git"
 billing_validation_repo_host: ""
 # Create or request a deploy token.
+# Reference: https://docs.gitlab.com/ee/user/project/deploy_tokens/
 billing_validation_repo_username: ""
 billing_validation_repo_token: ""
 
-# LRC Oracle billing database settings.
-# TODO: For LRC deployments with access to Oracle, set these.
+# TODO: For LRC deployments with access to the Oracle billing database, set
+# TODO: these.
 oracle_billing_db_dsn: ""
 oracle_billing_db_user: ""
 oracle_billing_db_passwd: ""
 
-# API settings.
+#------------------------------------------------------------------------------
+# Allowance renewal survey settings
+#------------------------------------------------------------------------------
+
+# # Backend options: 'google_forms', 'permissive'
+# renewal_survey_backend: 'google_forms'
+
+# # TODO: Uncomment only the section relevant to the specified renewal_survey_backend.
+
+# # Renewal survey settings: 'google_forms' backend.
+# renewal_survey_google_forms_service_account_credentials_file_path: '/path/to/google-service-account-key.json'
+# renewal_survey_google_forms_survey_data_file_path: '/path/to/google-forms-survey-data.json'
+# renewal_survey_google_forms_survey_data_cache_key: 'renewal_survey_google_forms_survey_data'
+
+# # Renewal survey settings: 'permissive' backend.
+# # N/A: No applicable settings.
+
+#------------------------------------------------------------------------------
+# REST API settings
+#------------------------------------------------------------------------------
+
 # If true, bypass all checks at job submission time.
 allow_all_jobs: false
 
+#------------------------------------------------------------------------------
+# Sentry settings
+#------------------------------------------------------------------------------
+
 # The URL of the Sentry instance to send errors to.
 sentry_dsn: ""
 
 ###############################################################################
-# staging_settings
+# Deployment-specific settings: staging, production, development
 ###############################################################################
 
+# TODO: Uncomment the section pertaining to the current deployment type.
+
+#------------------------------------------------------------------------------
+# staging_settings
+#------------------------------------------------------------------------------
+
 # # The type of deployment ('dev', 'prod', 'staging').
 # deployment: staging
 
 # # The user to run commands under and set permissions using.
 # djangooperator: root
 
 # # Whether to run the Django application in DEBUG mode.
+# # Note: Use uppercase True/False so that Python interprets this as a boolean.
 # debug: True
 
 # # The path to the parent directory containing the Git repository.
@@ -224,9 +309,9 @@ sentry_dsn: ""
 # # TODO: Set these addresses to yours.
 # request_approval_cc_list: []
 
-###############################################################################
+#------------------------------------------------------------------------------
 # prod_settings
-###############################################################################
+#------------------------------------------------------------------------------
 
 # # The type of deployment ('dev', 'prod', 'staging').
 # deployment: prod
@@ -235,6 +320,7 @@ sentry_dsn: ""
 # djangooperator: meli
 
 # # Whether to run the Django application in DEBUG mode.
+# # Note: Use uppercase True/False so that Python interprets this as a boolean.
 # debug: False
 
 # # The path to the parent directory containing the Git repository.
@@ -280,9 +366,9 @@ sentry_dsn: ""
 # # TODO: Set these addresses to yours.
 # request_approval_cc_list: []
 
-###############################################################################
+#------------------------------------------------------------------------------
 # dev_settings
-###############################################################################
+#------------------------------------------------------------------------------
 
 # # The type of deployment ('dev', 'prod', 'staging').
 # deployment: dev
@@ -291,6 +377,7 @@ sentry_dsn: ""
 # djangooperator: vagrant
 
 # # Whether to run the Django application in DEBUG mode.
+# # Note: Use uppercase True/False so that Python interprets this as a boolean.
 # debug: True
 
 # # The path to the parent directory containing the Git repository.