Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paver deprecation - attempt #? #138

Merged
merged 3 commits into from
Dec 18, 2024
Merged

Paver deprecation - attempt #? #138

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b9a69c2
Revert "Revert "Retry Paver Deprecation""
dianakhuang Dec 17, 2024
e8e4a68
fix: Remove unnecessary npm install call.
dianakhuang Dec 17, 2024
a1a1b23
fix: Force djangoapps to be loaded before we run webpack.
dianakhuang Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docker/build/edxapp/ansible_overrides.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ edxapp_npm_production: "no"

EDXAPP_LMS_GUNICORN_EXTRA_CONF: 'reload = True'

EDXAPP_NO_PREREQ_INSTALL: 0
COMMON_SSH_PASSWORD_AUTH: "yes"
EDXAPP_CMS_BASE: "edx.devstack.studio:18010"
EDXAPP_OAUTH_ENFORCE_SECURE: false
Expand Down
2 changes: 1 addition & 1 deletion playbooks/roles/aws_devstack/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
cd edx-repos/devstack/
make dev.pull.large-and-slow
make dev.up.lms
. /home/ubuntu/.profile && docker compose exec lms env TERM=xterm-256color bash -c 'make requirements && npm ci && paver update_assets && python manage.py lms migrate'
. /home/ubuntu/.profile && docker compose exec lms env TERM=xterm-256color bash -c 'make requirements && npm ci && npm run build && ./manage.py lms collectstatic --noinput && ./manage.py cms collectstatic --noinput && python manage.py lms migrate'
async: 1800
poll: 10
become_user: ubuntu
Expand Down
4 changes: 0 additions & 4 deletions playbooks/roles/edxapp/defaults/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -487,9 +487,6 @@ EDXAPP_CMS_STATIC_URL_BASE: "{{ EDXAPP_STATIC_URL_BASE }}"
# does not affect verified students
EDXAPP_PAID_COURSE_REGISTRATION_CURRENCY: ['usd', '$']

# Configure paver tasks in edx-platform to skip Python/Ruby/Node installation
EDXAPP_NO_PREREQ_INSTALL: 1

# whether to setup the python codejail or not
EDXAPP_PYTHON_SANDBOX: true
# this next setting, if true, turns on actual sandbox enforcement. If not true,
Expand Down Expand Up @@ -1176,7 +1173,6 @@ edxapp_helper_scripts:

edxapp_environment_default:
LANG: "{{ EDXAPP_LANG }}"
NO_PREREQ_INSTALL: "{{ EDXAPP_NO_PREREQ_INSTALL }}"
SKIP_WS_MIGRATIONS: 1
PATH: "{{ edxapp_deploy_path }}"
# the settings module for edxapp, DJANGO_SETTINGS_MODULE
Expand Down
9 changes: 0 additions & 9 deletions playbooks/roles/edxapp/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,6 @@
- devstack
- devstack:install

- name: add paver autocomplete to bashrc
lineinfile:
dest: /root/.bashrc
line: "source {{ edxapp_code_dir }}/scripts/paver_autocomplete.sh"
when: devstack is defined and devstack
tags:
- devstack
- devstack:install

- name: create edxapp log dir
file:
path: "{{ edxapp_log_dir }}"
Expand Down
4 changes: 2 additions & 2 deletions playbooks/roles/edxapp/tasks/service_variant_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,8 +312,8 @@
- gather_static_assets
- assets

# Gather assets using paver if possible
- name: "gather static assets with paver and pull translations"
# Build static assets if possible
- name: "gather static assets"
command: "{{ COMMON_BIN_DIR }}/edxapp-update-assets"
when: celery_worker is not defined and not devstack
tags:
Expand Down
25 changes: 22 additions & 3 deletions playbooks/roles/edxapp/templates/edx/bin/edxapp-update-assets.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,32 @@
{% include "edxapp_common.j2" %}

export WEBPACK_CONFIG_PATH=webpack.prod.config.js
export STATIC_ROOT_LMS=/edx/var/edxapp/staticfiles
export STATIC_ROOT_CMS=/edx/var/edxapp/staticfiles/studio
# This is necessary to properly build webworkers, which depends on this value being a dictionary.
export JS_ENV_EXTRA_CONFIG={}


# We need to make a call to a management command before running webpack because proctoring runs a webpack
# webworker that only gets built if the proctoring djangoapp writes out a `workers.json`:
# https://github.com/openedx/edx-proctoring/blob/73c7f55e2be91324fa07fec6e6ac0a667fdd8412/edx_proctoring/apps.py#L4
{% if edxapp_staticfiles_storage_overrides %}
{% for override in edxapp_staticfiles_storage_overrides %}
export STATICFILES_STORAGE={{ override | quote }}
sudo -E -H -u {{ edxapp_user }} \
env "PATH=$PATH" "STATICFILES_STORAGE={{ override }}" \
{{ edxapp_venv_bin }}/paver update_assets --debug-collect --settings=$EDX_PLATFORM_SETTINGS
env "PATH=$PATH" \
{{ edxapp_venv_bin }} python manage.py lms --settings=$EDX_PLATFORM_SETTINGS print_setting STATIC_ROOT WEBPACK_CONFIG_PATH \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call (or the other one) caused a very unhelpful error during the Ansible run: env: ‘/edx/app/edxapp/venvs/edxapp/bin’: Permission denied

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ohhh, I see the problem...

Suggested change
{{ edxapp_venv_bin }} python manage.py lms --settings=$EDX_PLATFORM_SETTINGS print_setting STATIC_ROOT WEBPACK_CONFIG_PATH \
{{ edxapp_venv_bin }}/python manage.py lms --settings=$EDX_PLATFORM_SETTINGS print_setting STATIC_ROOT WEBPACK_CONFIG_PATH \

&& npm run webpack \
&& npm run compile-sass -- --theme-dir /edx/var/edx-themes/edx-themes/edx-platform --theme-dir /edx/app/edxapp/edx-platform/themes \
&& {{ edxapp_venv_bin }}/python manage.py lms collectstatic --noinput --settings=$EDX_PLATFORM_SETTINGS \
&& {{ edxapp_venv_bin }}/python manage.py cms collectstatic --noinput --settings=$EDX_PLATFORM_SETTINGS
{% endfor %}
{% else %}
sudo -E -H -u {{ edxapp_user }} \
env "PATH=$PATH" \
{{ edxapp_venv_bin }}/paver update_assets --debug-collect --settings $EDX_PLATFORM_SETTINGS
{{ edxapp_venv_bin }} python manage.py lms --settings=$EDX_PLATFORM_SETTINGS print_setting STATIC_ROOT WEBPACK_CONFIG_PATH \
&& npm run webpack \
&& npm run compile-sass -- --theme-dir /edx/var/edx-themes/edx-themes/edx-platform --theme-dir /edx/app/edxapp/edx-platform/themes \
&& {{ edxapp_venv_bin }}/python manage.py lms collectstatic --noinput --settings=$EDX_PLATFORM_SETTINGS \
&& {{ edxapp_venv_bin }}/python manage.py cms collectstatic --noinput --settings=$EDX_PLATFORM_SETTINGS
{% endif %}
19 changes: 0 additions & 19 deletions playbooks/roles/test_build_server/defaults/main.yml
View file
Open in desktop

This file was deleted.

56 changes: 0 additions & 56 deletions playbooks/roles/test_build_server/files/test-development-environment.sh
View file
Open in desktop

This file was deleted.

26 changes: 0 additions & 26 deletions playbooks/roles/test_build_server/meta/main.yml
View file
Open in desktop

This file was deleted.

48 changes: 0 additions & 48 deletions playbooks/roles/test_build_server/tasks/main.yml
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion playbooks/vagrant-analytics.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
devstack: true
disable_edx_services: true
mongo_enable_journal: false
EDXAPP_NO_PREREQ_INSTALL: 0
COMMON_SSH_PASSWORD_AUTH: "yes"
EDXAPP_LMS_BASE: 127.0.0.1:8000
EDXAPP_OAUTH_ENFORCE_SECURE: false
Expand Down
10 changes: 5 additions & 5 deletions util/jenkins/app-container-provisioner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,10 @@ chown :www-data /edx/etc/${app_service_name}.yml

if [[ ${app_service_name} == 'lms' || ${app_service_name} == 'cms' ]]; then
# run migrations
docker run --network=host --rm -u='www-data' -e NO_PREREQ_INSTALL="1" -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} showmigrations --database default
docker run --network=host --rm -u='www-data' -e NO_PREREQ_INSTALL="1" -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} migrate --database default --noinput
docker run --network=host --rm -u='www-data' -e NO_PREREQ_INSTALL="1" -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} showmigrations --database student_module_history
docker run --network=host --rm -u='www-data' -e NO_PREREQ_INSTALL="1" -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} migrate --database student_module_history --noinput
docker run --network=host --rm -u='www-data' -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} showmigrations --database default
docker run --network=host --rm -u='www-data' -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} migrate --database default --noinput
docker run --network=host --rm -u='www-data' -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} showmigrations --database student_module_history
docker run --network=host --rm -u='www-data' -e SKIP_WS_MIGRATIONS="1" -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py ${app_service_name} migrate --database student_module_history --noinput
else
# Run app migrations
docker run --network=host --rm -u='www-data' -e ${app_cfg}=/edx/etc/${app_service_name}.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.settings.production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /edx/var/${app_name}:/edx/var/${app_name} -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest python3 manage.py migrate
Expand Down Expand Up @@ -123,7 +123,7 @@ fi
if [[ ${app_service_name} == 'lms' ]]; then
# temporary hack, create npm-install.log file
touch /edx/app/edxapp/edx-platform/test_root/log/npm-install.log
docker run --network=host --rm -u='root' -e NO_PREREQ_INSTALL="1" -e SKIP_WS_MIGRATIONS="1" -e LMS_CFG=/edx/etc/${app_service_name}.yml -e CMS_CFG=/edx/etc/cms.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /var/tmp/cms.yml:/edx/etc/cms.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /edx/var/${app_name}:/edx/var/${app_name} -v /edx/app/edxapp/edx-platform/test_root/log/npm-install.log:/edx/app/edxapp/edx-platform/test_root/log/npm-install.log -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest paver update_assets --debug-collect --settings=docker-production
docker run --network=host --rm -u='root' -e SKIP_WS_MIGRATIONS="1" -e LMS_CFG=/edx/etc/${app_service_name}.yml -e CMS_CFG=/edx/etc/cms.yml -e DJANGO_SETTINGS_MODULE=${app_service_name}.envs.docker-production -e SERVICE_VARIANT=${app_service_name} -e EDX_PLATFORM_SETTINGS=docker-production -e STATIC_ROOT_LMS=/edx/var/edxapp/staticfiles -e STATIC_ROOT_CMS=/edx/var/edxapp/staticfiles/studio -v /edx/etc/${app_service_name}.yml:/edx/etc/${app_service_name}.yml -v /var/tmp/cms.yml:/edx/etc/cms.yml -v /edx/var/edx-themes:/edx/var/edx-themes -v /edx/var/${app_name}:/edx/var/${app_name} -v /edx/app/edxapp/edx-platform/test_root/log/npm-install.log:/edx/app/edxapp/edx-platform/test_root/log/npm-install.log -v /var/run/mysqld/mysqld.sock:/var/run/mysqld/mysqld.sock ${app_repo}:latest npm run build && ./manage.py lms collectstatic --noinput --settings=docker-production && ./manage.py cms collectstatic --noinput --settings=docker-production
fi

# Generate docker-compose file for app service
Expand Down
Loading
Loading